Attachment 711787 Details for Bug 921643 – vdsm log

vdsm log

vdsm.log (text/plain), 14.99 MB, created by Pavel Zhukov on 2013-03-18 08:09:46 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Pavel Zhukov

Created: 2013-03-18 08:09:46 UTC

Size: 14.99 MB

patch

obsolete

>Thread-22::DEBUG::2013-03-17 04:01:05,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:01:05,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150135 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:01:07,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:01:07,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00289429 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:01:15,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:01:15,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124217 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:01:18,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:01:18,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083396 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:01:25,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:01:25,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141883 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:01:28,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:01:28,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000943156 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:01:35,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:01:35,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868915 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:01:38,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:01:38,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161839 s, 2.5 MB/s\n'; <rc> = 0
>Thread-13739::WARNING::2013-03-17 04:01:40,165::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:01:40,170::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 04:01:45,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:01:45,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00205929 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:01:48,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:01:48,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846208 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:01:50,173::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:01:50,174::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:01:50,626::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:01:50,632::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:01:50,634::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:01:55,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:01:55,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00259332 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:01:58,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:01:58,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793128 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:02:05,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:02:05,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00698415 s, 586 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:02:08,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:02:08,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160142 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:02:15,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:02:15,801::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109287 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:02:18,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:02:18,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893028 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:02:25,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:02:25,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870553 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:02:28,556::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:02:28,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000727921 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:02:35,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:02:35,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164943 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:02:38,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:02:38,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169867 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:02:45,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:02:45,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118409 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:02:48,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:02:48,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081496 s, 5.0 MB/s\n'; <rc> = 0
>Thread-13740::WARNING::2013-03-17 04:02:50,664::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:02:50,668::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 04:02:55,944::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:02:55,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000990561 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:02:58,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:02:58,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763532 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:03:00,670::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:03:00,671::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:03:01,106::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:03:01,112::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:03:01,114::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:03:05,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:03:06,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765679 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:03:08,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:03:08,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906884 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:03:16,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:03:16,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815002 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:03:18,938::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:03:18,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776467 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:03:26,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:03:26,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880762 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:03:29,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:03:29,040::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805781 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:03:36,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:03:36,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148695 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:03:39,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:03:39,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774566 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:03:46,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:03:46,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789524 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:03:49,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:03:49,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867913 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:03:56,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:03:56,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181396 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:03:59,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:03:59,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000699509 s, 5.9 MB/s\n'; <rc> = 0
>Thread-13741::WARNING::2013-03-17 04:04:01,139::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:04:01,142::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 04:04:06,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:04:06,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773743 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:04:09,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:04:09,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085845 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:04:11,144::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 04:04:11,145::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:04:11,146::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:04:11,538::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:04:11,545::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:04:11,546::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:04:16,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:04:16,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789897 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:04:19,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:04:19,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015324 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:04:26,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:04:26,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143275 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:04:29,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:04:29,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00195712 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:04:36,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:04:36,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101542 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:04:39,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:04:39,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011423 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:04:46,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:04:46,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840626 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:04:49,604::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 04:04:49,605::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 04:04:49,606::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:04:49,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000819676 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:04:49,635::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000819676 s, 2.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 04:04:49,636::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 04:04:49,636::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 04:04:49,638::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 04:04:49,639::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:04:49,639::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:04:49,640::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:04:49,640::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:04:49,641::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:04:49,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:04:50,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:04:50,047::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:04:50,048::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 04:04:50,051::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 04:04:50,053::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 04:04:50,054::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 04:04:50,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:04:50,413::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:04:50,415::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:04:50,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104176 s, 393 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:04:56,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:04:56,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165181 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:05:00,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:05:00,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838505 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:05:06,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:05:06,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878754 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:05:10,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:05:10,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799032 s, 5.1 MB/s\n'; <rc> = 0
>Thread-13742::WARNING::2013-03-17 04:05:11,580::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:05:11,583::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 04:05:16,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:05:16,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00067179 s, 6.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:05:20,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:05:21,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138925 s, 295 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:05:21,586::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:05:21,587::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:05:22,018::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:05:22,025::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:05:22,027::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:05:26,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:05:26,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117777 s, 348 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:05:31,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:05:31,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077429 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:05:36,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:05:36,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000951265 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:05:41,227::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:05:41,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153105 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:05:46,669::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 04:05:46,671::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 04:05:46,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:05:46,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000775943 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:05:46,697::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000775943 s, 2.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 04:05:46,697::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 04:05:46,697::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 04:05:46,700::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 04:05:46,701::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:05:46,702::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:05:46,702::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:05:46,703::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:05:46,703::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:05:46,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:05:47,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:05:47,141::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:05:47,142::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 04:05:47,144::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 04:05:47,145::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 04:05:47,146::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 04:05:47,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:05:47,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:05:47,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:05:47,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153489 s, 267 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:05:51,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:05:51,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166501 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:05:57,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:05:57,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011187 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:06:01,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:06:01,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159704 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:06:07,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:06:07,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164324 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:06:11,465::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:06:11,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150714 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:06:17,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:06:17,667::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817665 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:06:21,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:06:21,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082297 s, 5.0 MB/s\n'; <rc> = 0
>Thread-13743::WARNING::2013-03-17 04:06:22,067::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:06:22,075::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 04:06:27,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:06:27,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840484 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:06:31,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:06:31,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00206025 s, 2.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:06:32,081::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:06:32,082::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:06:32,549::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:06:32,556::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:06:32,557::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:06:37,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:06:37,742::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00607891 s, 674 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:06:41,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:06:41,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174255 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:06:47,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:06:47,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123806 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:06:52,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:06:52,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000987132 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:06:57,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:06:57,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166336 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:07:02,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:07:02,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171724 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:07:07,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:07:07,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163331 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:07:12,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:07:12,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016343 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:07:17,899::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:07:17,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000969469 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:07:22,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:07:22,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015585 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:07:27,950::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:07:27,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130079 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:07:32,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:07:32,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129952 s, 3.2 MB/s\n'; <rc> = 0
>Thread-13744::WARNING::2013-03-17 04:07:32,600::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:07:32,611::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 04:07:37,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:07:38,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856791 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:07:42,620::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:07:42,621::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:07:42,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:07:42,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00921858 s, 444 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:07:43,057::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:07:43,063::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:07:43,064::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:07:48,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:07:48,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00300384 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:07:52,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:07:52,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116148 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:07:58,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:07:58,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000993071 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:08:02,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:08:02,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884143 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:08:08,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:08:08,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904248 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:08:12,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:08:12,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809054 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:08:18,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:08:18,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00194 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:08:22,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:08:23,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156065 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:08:28,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:08:28,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00797888 s, 513 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:08:33,060::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:08:33,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133795 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:08:38,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:08:38,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00211777 s, 1.9 MB/s\n'; <rc> = 0
>Thread-13745::WARNING::2013-03-17 04:08:43,102::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:08:43,106::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:08:43,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:08:43,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822322 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:08:48,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:08:48,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016021 s, 2.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:08:53,108::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:08:53,109::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:08:53,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:08:53,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157772 s, 2.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:08:53,626::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:08:53,631::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:08:53,632::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:08:58,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:08:58,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102257 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:09:03,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:09:03,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805631 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:09:08,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:09:08,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00322795 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:09:13,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:09:13,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906338 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:09:18,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:09:18,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839339 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:09:23,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:09:23,484::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156406 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:09:28,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:09:28,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00906922 s, 452 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:09:33,528::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:09:33,562::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000691237 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:09:38,560::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:09:38,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764464 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:09:43,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:09:43,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106014 s, 386 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:09:48,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:09:48,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135798 s, 3.0 MB/s\n'; <rc> = 0
>Thread-13746::WARNING::2013-03-17 04:09:53,659::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:09:53,662::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:09:53,704::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 04:09:53,706::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 04:09:53,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:09:53,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000862365 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:09:53,736::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000862365 s, 2.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 04:09:53,737::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 04:09:53,737::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 04:09:53,739::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 04:09:53,740::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:09:53,740::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:09:53,741::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:09:53,741::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:09:53,742::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:09:53,742::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:09:54,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:09:54,223::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:09:54,224::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 04:09:54,226::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 04:09:54,227::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 04:09:54,228::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 04:09:54,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:09:54,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:09:54,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:09:54,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00877157 s, 467 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:09:58,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:09:58,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102271 s, 4.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:10:03,671::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 04:10:03,672::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:10:03,673::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:10:04,086::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:10:04,094::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:10:04,095::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:10:04,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:10:04,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125063 s, 328 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:10:08,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:10:08,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078588 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:10:14,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:10:14,820::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000648219 s, 6.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:10:18,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:10:18,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105164 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:10:24,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:10:24,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766943 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:10:28,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:10:28,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146302 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:10:34,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:10:34,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803047 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:10:38,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:10:38,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161796 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:10:45,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:10:45,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117013 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:10:48,981::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 04:10:48,983::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 04:10:48,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:10:49,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00107256 s, 1.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:10:49,023::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00107256 s, 1.9 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 04:10:49,023::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 04:10:49,023::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 04:10:49,025::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 04:10:49,026::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:10:49,026::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:10:49,027::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:10:49,027::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:10:49,028::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:10:49,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:10:49,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:10:49,461::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:10:49,462::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 04:10:49,465::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 04:10:49,466::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 04:10:49,466::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 04:10:49,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:10:49,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:10:49,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:10:49,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000943135 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:10:55,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:10:55,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168307 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:10:59,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:10:59,973::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001187 s, 3.5 MB/s\n'; <rc> = 0
>Thread-13747::WARNING::2013-03-17 04:11:04,132::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:11:04,136::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:11:05,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:11:05,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168114 s, 244 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:11:09,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:11:10,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113453 s, 3.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:11:14,138::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:11:14,140::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:11:14,546::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:11:14,553::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:11:14,554::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:11:15,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:11:15,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00870849 s, 470 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:11:20,021::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:11:20,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787623 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:11:25,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:11:25,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820462 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:11:30,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:11:30,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812879 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:11:35,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:11:35,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154923 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:11:40,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:11:40,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844588 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:11:45,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:11:45,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000957883 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:11:50,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:11:50,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151502 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:11:55,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:11:55,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884213 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:12:00,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:12:00,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781538 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:12:05,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:12:05,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762374 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:12:10,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:12:10,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876586 s, 4.7 MB/s\n'; <rc> = 0
>Thread-13748::WARNING::2013-03-17 04:12:14,580::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:12:14,583::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:12:15,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:12:15,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113943 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:12:20,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:12:20,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834706 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:12:24,586::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:12:24,587::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:12:25,004::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:12:25,011::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:12:25,012::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:12:25,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:12:25,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124462 s, 329 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:12:30,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:12:30,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866394 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:12:35,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:12:35,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796822 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:12:40,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:12:40,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145983 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:12:45,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:12:45,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791024 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:12:50,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:12:50,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881904 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:12:56,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:12:56,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142924 s, 287 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:13:00,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:13:00,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164918 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:13:06,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:13:06,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00180319 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:13:10,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:13:10,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164205 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:13:16,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:13:16,260::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00204267 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:13:20,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:13:20,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150678 s, 2.7 MB/s\n'; <rc> = 0
>Thread-13749::WARNING::2013-03-17 04:13:25,040::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:13:25,045::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:13:26,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:13:26,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898275 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:13:30,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:13:30,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00231114 s, 1.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:13:35,049::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:13:35,050::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:13:35,469::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:13:35,476::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:13:35,477::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:13:36,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:13:36,439::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0318615 s, 129 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:13:40,617::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:13:40,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000681067 s, 6.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:13:46,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:13:46,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141691 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:13:50,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:13:50,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016087 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:13:56,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:13:56,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774607 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:14:00,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:14:00,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787392 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:14:06,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:14:06,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088529 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:14:10,738::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:14:10,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124068 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:14:16,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:14:16,738::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848777 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:14:20,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:14:20,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149196 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:14:26,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:14:26,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878367 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:14:30,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:14:30,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156641 s, 2.6 MB/s\n'; <rc> = 0
>Thread-13750::WARNING::2013-03-17 04:14:35,507::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:14:35,510::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:14:36,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:14:36,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764119 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:14:40,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:14:40,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788868 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:14:45,512::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:14:45,513::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:14:45,944::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:14:45,952::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:14:45,954::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:14:46,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:14:46,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010252 s, 400 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:14:50,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:14:50,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919037 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:14:57,015::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 04:14:57,017::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 04:14:57,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:14:57,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000880181 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:14:57,051::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000880181 s, 2.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 04:14:57,051::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 04:14:57,051::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 04:14:57,053::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 04:14:57,054::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:14:57,055::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:14:57,055::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:14:57,055::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:14:57,056::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:14:57,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:14:57,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:14:57,478::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:14:57,479::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 04:14:57,481::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 04:14:57,483::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 04:14:57,484::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 04:14:57,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:14:57,912::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:14:57,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:14:57,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132171 s, 310 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:15:00,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:15:00,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000717816 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:15:08,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:15:08,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851593 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:15:11,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:15:11,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138581 s, 296 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:15:18,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:15:18,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765069 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:15:21,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:15:21,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091068 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:15:28,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:15:28,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142549 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:15:31,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:15:31,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106855 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:15:38,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:15:38,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160225 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:15:41,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:15:41,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160024 s, 2.6 MB/s\n'; <rc> = 0
>Thread-13751::WARNING::2013-03-17 04:15:45,981::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:15:45,985::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:15:48,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:15:48,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858002 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:15:51,239::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 04:15:51,241::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 04:15:51,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:15:51,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0221263 s, 92.6 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:15:51,302::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0221263 s, 92.6 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 04:15:51,302::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 04:15:51,302::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 04:15:51,305::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 04:15:51,306::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:15:51,307::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:15:51,307::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:15:51,307::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:15:51,308::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:15:51,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:15:51,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:15:51,726::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:15:51,732::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 04:15:51,735::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 04:15:51,736::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 04:15:51,737::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 04:15:51,738::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:15:52,123::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:15:52,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:15:52,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171976 s, 238 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:15:55,986::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 04:15:55,987::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:15:55,988::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:15:56,419::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:15:56,425::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:15:56,427::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:15:58,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:15:58,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139506 s, 294 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:16:02,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:16:02,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851534 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:16:08,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:16:08,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764694 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:16:12,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:16:12,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808393 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:16:18,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:16:18,569::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793406 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:16:22,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:16:22,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771134 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:16:28,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:16:28,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765498 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:16:32,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:16:32,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795232 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:16:38,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:16:38,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748681 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:16:42,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:16:42,365::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118422 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:16:48,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:16:48,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101941 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:16:52,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:16:52,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151807 s, 2.7 MB/s\n'; <rc> = 0
>Thread-13752::WARNING::2013-03-17 04:16:56,453::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:16:56,466::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:16:58,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:16:58,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000706602 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:17:02,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:17:02,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154846 s, 2.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:17:06,474::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:17:06,475::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:17:06,948::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:17:06,956::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:17:06,957::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:17:08,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:17:08,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104965 s, 390 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:17:12,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:17:12,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155805 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:17:18,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:17:19,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083257 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:17:22,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:17:22,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000958058 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:17:29,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:17:29,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154221 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:17:32,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:17:32,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001222 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:17:39,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:17:39,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834324 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:17:42,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:17:42,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078927 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:17:49,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:17:49,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145032 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:17:52,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:17:52,649::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153244 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:17:59,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:17:59,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107673 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:18:02,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:18:02,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164142 s, 2.5 MB/s\n'; <rc> = 0
>Thread-13753::WARNING::2013-03-17 04:18:06,985::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:18:06,993::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:18:09,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:18:09,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162599 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:18:12,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:18:12,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841607 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:18:16,995::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:18:16,997::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:18:17,432::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:18:17,440::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:18:17,441::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:18:19,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:18:19,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00351068 s, 1.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:18:22,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:18:22,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154804 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:18:29,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:18:29,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00069531 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:18:32,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:18:32,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142619 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:18:39,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:18:39,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861437 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:18:42,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:18:42,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173544 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:18:49,685::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:18:49,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157678 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:18:52,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:18:52,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156994 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:18:59,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:18:59,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808439 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:19:02,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:19:02,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143034 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:19:09,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:19:09,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083996 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:19:12,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:19:13,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141629 s, 2.9 MB/s\n'; <rc> = 0
>Thread-13754::WARNING::2013-03-17 04:19:17,477::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:19:17,482::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:19:19,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:19:19,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000965632 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:19:23,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:19:23,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825389 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:19:27,490::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:19:27,491::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:19:27,904::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:19:27,911::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:19:27,912::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:19:29,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:19:30,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124694 s, 328 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:19:33,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:19:33,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151338 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:19:40,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:19:40,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826797 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:19:43,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:19:43,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155408 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:19:50,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:19:50,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161043 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:19:53,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:19:53,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872229 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:20:00,227::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 04:20:00,229::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 04:20:00,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:20:00,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000828647 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:20:00,259::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000828647 s, 2.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 04:20:00,259::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 04:20:00,259::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 04:20:00,261::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 04:20:00,262::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:20:00,263::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:20:00,263::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:20:00,264::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:20:00,264::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:20:00,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:20:00,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:20:00,674::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:20:00,675::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 04:20:00,678::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 04:20:00,679::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 04:20:00,679::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 04:20:00,681::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:20:01,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:20:01,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:20:01,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179323 s, 228 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:20:03,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:20:03,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836409 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:20:11,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:20:11,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080803 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:20:13,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:20:13,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759148 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:20:21,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:20:21,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166941 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:20:23,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:20:23,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152461 s, 2.7 MB/s\n'; <rc> = 0
>Thread-13755::WARNING::2013-03-17 04:20:27,939::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:20:27,942::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:20:31,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:20:31,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749171 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:20:33,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:20:33,365::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016719 s, 2.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:20:37,948::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:20:37,949::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:20:38,408::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:20:38,415::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:20:38,416::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:20:41,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:20:41,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00889916 s, 460 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:20:43,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:20:43,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0211167 s, 194 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:20:51,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:20:51,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752935 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:20:53,427::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 04:20:53,429::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 04:20:53,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:20:53,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000750872 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:20:53,457::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000750872 s, 2.7 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 04:20:53,457::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 04:20:53,458::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 04:20:53,459::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 04:20:53,464::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:20:53,465::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:20:53,465::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:20:53,466::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:20:53,466::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:20:53,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:20:53,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:20:53,900::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:20:53,901::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 04:20:53,906::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 04:20:53,908::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 04:20:53,909::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 04:20:53,910::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:20:54,266::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:20:54,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:20:54,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778017 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:21:01,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:21:01,602::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789532 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:21:04,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:21:04,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880036 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:21:11,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:21:11,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785105 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:21:14,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:21:14,364::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079392 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:21:21,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:21:21,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140721 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:21:24,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:21:24,402::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143537 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:21:31,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:21:31,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140278 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:21:34,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:21:34,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778864 s, 5.3 MB/s\n'; <rc> = 0
>Thread-13756::WARNING::2013-03-17 04:21:38,445::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:21:38,452::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:21:41,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:21:41,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832738 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:21:44,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:21:44,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831497 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:21:48,460::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 04:21:48,461::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:21:48,462::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:21:48,917::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:21:48,924::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:21:48,925::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:21:51,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:21:52,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00985833 s, 415 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:21:54,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:21:54,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00069753 s, 5.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:22:02,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:22:02,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180891 s, 226 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:22:04,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:22:04,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893408 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:22:12,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:22:12,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786524 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:22:14,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:22:14,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154196 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:22:22,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:22:22,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110686 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:22:24,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:22:24,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103389 s, 396 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:22:32,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:22:32,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154412 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:22:34,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:22:34,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825377 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:22:42,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:22:42,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00182695 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:22:44,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:22:44,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157007 s, 2.6 MB/s\n'; <rc> = 0
>Thread-13757::WARNING::2013-03-17 04:22:48,967::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:22:48,970::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:22:52,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:22:52,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0027499 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:22:54,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:22:54,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000720306 s, 5.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:22:58,973::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:22:58,975::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:22:59,492::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:22:59,499::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:22:59,501::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:23:02,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:23:02,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133835 s, 306 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:23:04,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:23:04,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159993 s, 256 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:23:12,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:23:12,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163418 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:23:14,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:23:14,880::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081694 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:23:22,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:23:22,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00369184 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:23:24,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:23:24,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888577 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:23:32,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:23:32,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154474 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:23:34,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:23:34,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00238152 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:23:42,857::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:23:42,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0007987 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:23:44,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:23:45,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847595 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:23:52,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:23:52,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0283431 s, 145 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:23:55,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:23:55,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177929 s, 2.3 MB/s\n'; <rc> = 0
>Thread-13758::WARNING::2013-03-17 04:23:59,533::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:23:59,537::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:24:03,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:24:03,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790572 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:24:05,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:24:05,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00830474 s, 493 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:24:09,545::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:24:09,547::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:24:10,066::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:24:10,072::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:24:10,074::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:24:13,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:24:13,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161805 s, 253 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:24:15,099::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:24:15,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.035832 s, 114 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:24:23,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:24:23,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762609 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:24:25,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:24:25,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000967734 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:24:33,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:24:33,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774338 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:24:35,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:24:35,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085706 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:24:43,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:24:43,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016359 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:24:45,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:24:45,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172017 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:24:53,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:24:53,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148675 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:24:55,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:24:55,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153789 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:25:03,504::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 04:25:03,505::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 04:25:03,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:25:03,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000760176 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:25:03,545::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000760176 s, 2.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 04:25:03,546::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 04:25:03,546::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 04:25:03,548::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 04:25:03,550::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:25:03,551::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:25:03,551::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:25:03,553::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:25:03,553::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:25:03,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:25:03,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:25:04,003::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:25:04,004::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 04:25:04,007::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 04:25:04,008::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 04:25:04,009::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 04:25:04,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:25:04,398::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:25:04,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:25:04,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101622 s, 403 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:25:05,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:25:05,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183608 s, 223 kB/s\n'; <rc> = 0
>Thread-13759::WARNING::2013-03-17 04:25:10,103::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:25:10,105::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:25:14,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:25:14,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00291446 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:25:15,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:25:15,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167137 s, 245 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:25:20,107::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:25:20,108::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:25:20,611::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:25:20,618::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:25:20,619::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:25:24,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:25:24,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0260636 s, 157 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:25:25,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:25:25,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167028 s, 245 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:25:34,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:25:34,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154779 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:25:35,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:25:35,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00531641 s, 770 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:25:44,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:25:44,742::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784271 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:25:45,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:25:45,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159558 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:25:54,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:25:54,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161694 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:25:55,627::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 04:25:55,629::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 04:25:55,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:25:55,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000880962 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:25:55,671::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000880962 s, 2.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 04:25:55,671::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 04:25:55,671::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 04:25:55,673::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 04:25:55,674::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:25:55,674::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:25:55,675::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:25:55,675::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:25:55,676::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:25:55,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:25:56,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:25:56,125::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:25:56,126::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 04:25:56,128::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 04:25:56,129::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 04:25:56,130::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 04:25:56,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:25:56,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:25:56,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:25:56,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126071 s, 325 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:26:04,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:26:04,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854461 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:26:06,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:26:06,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00314746 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:26:14,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:26:14,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831249 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:26:16,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:26:16,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141554 s, 2.9 MB/s\n'; <rc> = 0
>Thread-13760::WARNING::2013-03-17 04:26:20,664::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:26:20,666::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:26:25,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:26:25,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861434 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:26:26,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:26:26,742::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167712 s, 2.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:26:30,669::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:26:30,670::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:26:31,200::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:26:31,206::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:26:31,208::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:26:35,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:26:35,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00996745 s, 411 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:26:36,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:26:36,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101084 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:26:45,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:26:45,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762126 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:26:46,782::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:26:46,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755375 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:26:55,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:26:55,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014547 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:26:56,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:26:56,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770018 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:27:05,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:27:05,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156372 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:27:06,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:27:06,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106681 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:27:15,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:27:15,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154461 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:27:16,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:27:16,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147661 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:27:25,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:27:25,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123435 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:27:26,959::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:27:26,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156665 s, 2.6 MB/s\n'; <rc> = 0
>Thread-13761::WARNING::2013-03-17 04:27:31,238::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:27:31,245::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:27:35,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:27:35,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805967 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:27:37,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:27:37,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803138 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:27:41,247::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 04:27:41,249::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:27:41,249::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:27:41,720::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:27:41,726::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:27:41,728::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:27:45,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:27:45,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000931784 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:27:47,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:27:47,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829248 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:27:55,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:27:55,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810261 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:27:57,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:27:57,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867711 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:28:05,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:28:05,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000745742 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:28:07,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:28:07,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125304 s, 327 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:28:15,857::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:28:15,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805323 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:28:17,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:28:17,222::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755386 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:28:25,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:28:25,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924782 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:28:27,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:28:27,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157577 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:28:36,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:28:36,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737515 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:28:37,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:28:37,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849995 s, 4.8 MB/s\n'; <rc> = 0
>Thread-13762::WARNING::2013-03-17 04:28:41,754::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:28:41,758::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:28:46,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:28:46,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773813 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:28:47,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:28:47,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0205187 s, 200 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:28:51,760::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:28:51,761::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:28:52,218::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:28:52,225::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:28:52,226::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:28:56,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:28:56,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014455 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:28:57,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:28:57,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000719374 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:29:06,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:29:06,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792045 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:29:07,415::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:29:07,456::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841783 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:29:16,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:29:16,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765766 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:29:17,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:29:17,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00195561 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:29:26,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:29:26,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001532 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:29:27,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:29:27,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00180861 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:29:36,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:29:36,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892302 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:29:37,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:29:37,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000970595 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:29:46,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:29:46,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013495 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:29:47,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:29:47,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150132 s, 2.7 MB/s\n'; <rc> = 0
>Thread-13763::WARNING::2013-03-17 04:29:52,253::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:29:52,257::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:29:56,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:29:56,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892789 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:29:57,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:29:57,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153225 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:30:02,259::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:30:02,260::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:30:02,725::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:30:02,732::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:30:02,733::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:30:06,672::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 04:30:06,675::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 04:30:06,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:30:06,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000648309 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:30:06,708::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000648309 s, 3.2 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 04:30:06,708::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 04:30:06,708::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 04:30:06,710::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 04:30:06,711::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:30:06,712::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:30:06,712::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:30:06,712::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:30:06,713::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:30:06,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:30:07,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:30:07,164::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:30:07,165::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 04:30:07,167::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 04:30:07,168::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 04:30:07,169::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 04:30:07,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:30:07,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:30:07,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:30:07,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115135 s, 356 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:30:07,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:30:07,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163517 s, 250 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:30:17,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:30:17,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0289652 s, 141 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:30:17,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:30:17,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0403156 s, 102 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:30:27,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:30:28,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00685884 s, 597 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:30:28,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:30:28,060::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00234049 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:30:38,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:30:38,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847187 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:30:38,099::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:30:38,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160215 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:30:48,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:30:48,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103916 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:30:48,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:30:48,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000689263 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:30:58,124::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 04:30:58,126::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 04:30:58,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:30:58,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00107514 s, 1.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:30:58,161::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00107514 s, 1.9 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 04:30:58,161::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 04:30:58,162::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 04:30:58,163::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 04:30:58,164::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:30:58,165::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:30:58,166::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:30:58,167::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:30:58,167::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:30:58,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:30:58,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:30:58,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00604239 s, 678 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:30:58,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:30:58,650::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:30:58,652::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 04:30:58,655::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 04:30:58,656::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 04:30:58,656::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 04:30:58,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:30:59,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:30:59,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:30:59,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171827 s, 238 kB/s\n'; <rc> = 0
>Thread-13764::WARNING::2013-03-17 04:31:02,766::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:31:02,771::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:31:08,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:31:08,399::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000905618 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:31:09,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:31:09,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778991 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:31:12,773::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:31:12,774::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:31:13,229::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:31:13,235::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:31:13,237::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:31:18,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:31:18,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000695039 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:31:19,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:31:19,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138729 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:31:28,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:31:28,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823113 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:31:29,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:31:29,311::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000955786 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:31:38,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:31:38,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738347 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:31:39,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:31:39,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167653 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:31:48,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:31:48,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159397 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:31:49,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:31:49,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00195311 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:31:58,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:31:58,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148725 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:31:59,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:31:59,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919484 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:32:08,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:32:08,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012098 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:32:09,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:32:09,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781418 s, 5.2 MB/s\n'; <rc> = 0
>Thread-13765::WARNING::2013-03-17 04:32:13,260::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:32:13,264::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:32:18,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:32:18,944::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165674 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:32:19,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:32:19,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798083 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:32:23,266::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:32:23,267::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:32:23,825::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:32:23,832::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:32:23,833::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:32:28,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:32:29,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0270155 s, 152 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:32:29,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:32:29,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755479 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:32:39,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:32:39,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000956056 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:32:39,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:32:39,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882676 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:32:49,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:32:49,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842402 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:32:49,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:32:49,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846488 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:32:59,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:32:59,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140595 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:32:59,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:32:59,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776564 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:33:09,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:33:09,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862993 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:33:09,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:33:09,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102371 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:33:19,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:33:19,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177753 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:33:19,730::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:33:19,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150895 s, 2.7 MB/s\n'; <rc> = 0
>Thread-13766::WARNING::2013-03-17 04:33:23,859::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:33:23,863::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:33:29,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:33:29,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000719233 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:33:29,782::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:33:29,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932395 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:33:33,864::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 04:33:33,866::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:33:33,867::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:33:34,325::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:33:34,331::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:33:34,333::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:33:39,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:33:39,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165622 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:33:39,814::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:33:39,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746861 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:33:49,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:33:49,681::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00305025 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:33:49,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:33:49,880::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166781 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:33:59,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:33:59,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00332577 s, 1.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:33:59,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:33:59,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156411 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:34:09,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:34:09,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179994 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:34:09,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:34:09,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00073343 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:34:19,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:34:19,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752259 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:34:19,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:34:20,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017179 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:34:30,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:34:30,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:34:30,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011353 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:34:30,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115339 s, 3.6 MB/s\n'; <rc> = 0
>Thread-13767::WARNING::2013-03-17 04:34:34,359::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:34:34,363::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 04:34:40,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:34:40,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909307 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:34:40,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:34:40,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756458 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:34:44,365::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:34:44,366::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:34:44,805::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:34:44,811::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:34:44,813::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:34:50,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:34:50,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758266 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:34:50,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:34:50,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862714 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:35:00,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:35:00,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149158 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:35:00,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:35:00,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801476 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:35:10,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:35:10,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744988 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:35:10,423::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 04:35:10,425::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 04:35:10,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:35:10,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00135822 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:35:10,472::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00135822 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 04:35:10,472::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 04:35:10,473::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 04:35:10,474::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 04:35:10,475::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:35:10,476::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:35:10,476::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:35:10,477::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:35:10,477::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:35:10,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:35:10,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:35:10,916::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:35:10,917::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 04:35:10,920::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 04:35:10,921::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 04:35:10,921::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 04:35:10,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:35:11,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:35:11,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:35:11,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00657439 s, 623 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:35:20,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:35:20,404::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00070917 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:35:21,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:35:21,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111755 s, 367 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:35:30,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:35:30,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153527 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:35:31,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:35:31,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172935 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:35:40,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:35:40,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00261049 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:35:41,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:35:41,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151238 s, 2.7 MB/s\n'; <rc> = 0
>Thread-13768::WARNING::2013-03-17 04:35:44,839::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:35:44,842::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 04:35:50,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:35:50,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766242 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:35:51,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:35:51,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772731 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:35:54,844::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:35:54,846::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:35:55,305::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:35:55,312::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:35:55,313::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:36:00,554::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 04:36:00,556::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 04:36:00,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:36:00,591::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00108615 s, 1.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:36:00,595::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00108615 s, 1.9 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 04:36:00,596::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 04:36:00,596::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 04:36:00,597::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 04:36:00,598::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:36:00,599::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:36:00,600::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:36:00,600::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:36:00,600::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:36:00,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:36:01,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:36:01,035::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:36:01,036::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 04:36:01,039::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 04:36:01,046::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 04:36:01,046::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 04:36:01,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:36:01,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:36:01,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:36:01,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0389538 s, 105 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:36:01,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:36:01,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00877542 s, 467 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:36:11,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:36:11,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00326238 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:36:11,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:36:11,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00296738 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:36:21,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:36:21,658::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159175 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:36:21,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:36:21,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777451 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:36:31,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:36:31,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738673 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:36:31,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:36:31,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091579 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:36:41,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:36:41,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897029 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:36:42,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:36:42,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015174 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:36:51,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:36:51,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120473 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:36:52,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:36:52,132::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000989383 s, 4.1 MB/s\n'; <rc> = 0
>Thread-13769::WARNING::2013-03-17 04:36:55,339::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:36:55,345::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 04:37:01,782::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:37:01,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000978648 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:37:02,178::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:37:02,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909809 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:37:05,349::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:37:05,350::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:37:05,794::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:37:05,800::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:37:05,802::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:37:11,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:37:11,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00211713 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:37:12,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:37:12,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000927266 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:37:21,871::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:37:21,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860367 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:37:22,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:37:22,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789752 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:37:31,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:37:31,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803088 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:37:32,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:37:32,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00190071 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:37:41,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:37:41,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000734364 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:37:42,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:37:42,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156472 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:37:51,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:37:52,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00373678 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:37:52,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:37:52,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153564 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:38:02,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:38:02,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000944041 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:38:02,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:38:02,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00276086 s, 1.5 MB/s\n'; <rc> = 0
>Thread-13770::WARNING::2013-03-17 04:38:05,834::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:38:05,838::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 04:38:12,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:38:12,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091984 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:38:12,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:38:12,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780474 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:38:15,841::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:38:15,843::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:38:16,292::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:38:16,300::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:38:16,301::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:38:22,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:38:22,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150605 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:38:22,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:38:22,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101229 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:38:32,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:38:32,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132712 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:38:32,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:38:32,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016343 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:38:42,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:38:42,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158658 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:38:42,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:38:43,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00244195 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:38:52,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:38:52,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897345 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:38:53,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:38:53,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800763 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:39:02,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:39:02,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145754 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:39:03,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:39:03,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159616 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:39:12,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:39:12,392::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150083 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:39:13,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:39:13,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155942 s, 2.6 MB/s\n'; <rc> = 0
>Thread-13771::WARNING::2013-03-17 04:39:16,325::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:39:16,331::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 04:39:22,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:39:22,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867931 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:39:23,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:39:23,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793483 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:39:26,332::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 04:39:26,333::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:39:26,334::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:39:26,852::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:39:26,858::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:39:26,860::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:39:32,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:39:32,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841982 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:39:33,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:39:33,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811922 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:39:42,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:39:42,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861511 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:39:43,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:39:43,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173038 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:39:52,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:39:52,548::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153693 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:39:53,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:39:53,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775245 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:40:02,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:40:02,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752833 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:40:03,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:40:03,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016379 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:40:12,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:40:12,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164795 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:40:13,657::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 04:40:13,658::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 04:40:13,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:40:13,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00137542 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:40:13,702::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00137542 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 04:40:13,703::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 04:40:13,703::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 04:40:13,705::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 04:40:13,706::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:40:13,706::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:40:13,707::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:40:13,707::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:40:13,708::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:40:13,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:40:14,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:40:14,137::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:40:14,138::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 04:40:14,141::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 04:40:14,142::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 04:40:14,142::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 04:40:14,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:40:14,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:40:14,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:40:14,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133327 s, 307 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:40:22,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:40:22,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00348758 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:40:24,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:40:24,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0231186 s, 177 kB/s\n'; <rc> = 0
>Thread-13772::WARNING::2013-03-17 04:40:26,885::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:40:26,888::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 04:40:32,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:40:32,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081397 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:40:34,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:40:34,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170875 s, 2.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:40:36,891::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:40:36,893::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:40:37,331::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:40:37,337::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:40:37,339::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:40:42,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:40:42,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00381247 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:40:44,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:40:44,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008675 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:40:52,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:40:52,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166379 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:40:54,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:40:54,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159047 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:41:02,816::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 04:41:02,817::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 04:41:02,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:41:02,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00138534 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:41:02,873::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00138534 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 04:41:02,874::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 04:41:02,874::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 04:41:02,878::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 04:41:02,879::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:41:02,880::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:41:02,880::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:41:02,881::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:41:02,881::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:41:02,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:41:03,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:41:03,321::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:41:03,322::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 04:41:03,325::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 04:41:03,326::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 04:41:03,326::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 04:41:03,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:41:03,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:41:03,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:41:03,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101745 s, 403 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:41:04,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:41:05,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00829282 s, 494 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:41:13,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:41:13,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834389 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:41:15,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:41:15,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176242 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:41:23,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:41:23,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152717 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:41:25,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:41:25,222::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100718 s, 407 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:41:33,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:41:33,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142794 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:41:35,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:41:35,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000739262 s, 5.5 MB/s\n'; <rc> = 0
>Thread-13773::WARNING::2013-03-17 04:41:37,367::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:41:37,372::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 04:41:43,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:41:43,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901255 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:41:45,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:41:45,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0009142 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:41:47,376::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:41:47,377::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:41:47,988::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:41:47,996::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:41:47,997::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:41:53,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:41:53,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088526 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:41:55,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:41:55,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857084 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:42:03,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:42:04,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146047 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:42:05,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:42:05,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147946 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:42:14,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:42:14,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:42:15,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:42:15,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808968 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:42:24,045::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:42:24,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159304 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:42:25,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:42:25,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188798 s, 217 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:42:34,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:42:34,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152172 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:42:35,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:42:35,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775988 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:42:44,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:42:44,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149596 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:42:45,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:42:45,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162776 s, 252 kB/s\n'; <rc> = 0
>Thread-13774::WARNING::2013-03-17 04:42:48,061::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:42:48,064::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 04:42:54,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:42:54,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917734 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:42:55,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:42:55,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807764 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:42:58,066::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:42:58,067::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:42:58,546::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:42:58,565::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:42:58,567::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:43:04,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:43:04,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014825 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:43:05,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:43:05,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834176 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:43:14,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:43:14,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178783 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:43:16,034::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:43:16,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862692 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:43:24,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:43:24,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768652 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:43:26,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:43:26,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145852 s, 281 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:43:34,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:43:34,398::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134551 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:43:36,194::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:43:36,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773438 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:43:44,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:43:44,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911673 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:43:46,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:43:46,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153972 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:43:54,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:43:54,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784909 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:43:56,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:43:56,398::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132528 s, 3.1 MB/s\n'; <rc> = 0
>Thread-13775::WARNING::2013-03-17 04:43:58,591::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:43:58,594::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 04:44:04,482::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:44:04,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835483 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:44:06,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:44:06,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000710649 s, 5.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:44:08,596::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:44:08,597::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:44:09,040::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:44:09,046::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:44:09,047::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:44:14,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:44:14,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081341 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:44:16,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:44:16,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154934 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:44:24,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:44:24,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822179 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:44:26,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:44:26,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00099408 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:44:34,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:44:34,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00224153 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:44:36,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:44:36,712::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00183416 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:44:44,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:44:44,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000922018 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:44:46,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:44:46,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829841 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:44:54,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:44:54,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173241 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:44:56,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:44:56,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110629 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:45:04,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:45:04,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00917964 s, 446 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:45:06,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:45:06,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124055 s, 3.3 MB/s\n'; <rc> = 0
>Thread-13776::WARNING::2013-03-17 04:45:09,074::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:45:09,078::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 04:45:14,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:45:14,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154279 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:45:17,035::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 04:45:17,037::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 04:45:17,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:45:17,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00104436 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:45:17,075::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00104436 s, 2.0 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 04:45:17,075::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 04:45:17,076::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 04:45:17,078::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 04:45:17,079::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:45:17,080::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:45:17,081::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:45:17,081::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:45:17,082::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:45:17,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:45:17,494::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:45:17,499::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:45:17,500::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 04:45:17,503::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 04:45:17,504::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 04:45:17,505::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 04:45:17,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:45:17,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:45:17,871::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:45:17,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189602 s, 216 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:45:19,082::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 04:45:19,084::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:45:19,085::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:45:19,493::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:45:19,499::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:45:19,501::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:45:24,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:45:24,843::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0077207 s, 531 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:45:27,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:45:28,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117269 s, 349 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:45:34,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:45:34,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014644 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:45:38,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:45:38,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916421 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:45:44,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:45:44,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150822 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:45:48,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:45:48,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859078 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:45:54,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:45:54,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161547 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:45:58,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:45:58,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159695 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:46:04,997::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 04:46:04,999::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 04:46:04,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:46:05,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0013691 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:46:05,046::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0013691 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 04:46:05,047::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 04:46:05,047::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 04:46:05,051::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 04:46:05,053::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:46:05,053::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:46:05,054::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:46:05,054::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:46:05,054::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:46:05,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:46:05,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:46:05,462::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:46:05,463::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 04:46:05,465::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 04:46:05,466::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 04:46:05,467::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 04:46:05,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:46:05,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:46:05,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:46:05,853::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746001 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:46:08,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:46:08,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00816758 s, 501 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:46:15,859::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:46:15,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144144 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:46:18,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:46:18,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00441303 s, 928 kB/s\n'; <rc> = 0
>Thread-13777::WARNING::2013-03-17 04:46:19,545::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:46:19,548::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 04:46:25,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:46:25,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821836 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:46:28,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:46:28,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00521841 s, 785 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:46:29,550::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:46:29,551::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:46:30,012::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:46:30,019::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:46:30,020::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:46:35,941::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:46:35,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160957 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:46:38,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:46:38,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156413 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:46:45,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:46:46,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177675 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:46:48,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:46:48,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082321 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:46:56,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:46:56,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000687926 s, 6.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:46:58,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:46:58,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805319 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:47:06,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:47:06,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181934 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:47:08,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:47:08,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748273 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:47:16,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:47:16,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769317 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:47:18,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:47:18,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851691 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:47:26,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:47:26,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129385 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:47:28,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:47:29,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00844909 s, 485 kB/s\n'; <rc> = 0
>Thread-13778::WARNING::2013-03-17 04:47:30,050::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:47:30,054::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 04:47:36,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:47:36,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155982 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:47:39,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:47:39,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863558 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:47:40,056::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:47:40,057::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:47:40,490::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:47:40,498::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:47:40,499::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:47:46,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:47:46,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000934371 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:47:49,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:47:49,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154613 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:47:56,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:47:56,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156424 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:47:59,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:47:59,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915401 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:48:06,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:48:06,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837633 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:48:09,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:48:09,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00093918 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:48:16,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:48:16,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00182235 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:48:19,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:48:19,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101738 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:48:26,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:48:26,481::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139448 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:48:29,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:48:29,552::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181735 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:48:36,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:48:36,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154499 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:48:39,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:48:39,742::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178996 s, 2.3 MB/s\n'; <rc> = 0
>Thread-13779::WARNING::2013-03-17 04:48:40,529::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:48:40,536::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 04:48:46,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:48:46,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780427 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:48:50,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:48:50,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819264 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:48:50,539::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:48:50,540::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:48:50,943::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:48:50,950::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:48:50,951::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:48:56,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:48:56,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888862 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:49:00,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:49:00,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000971231 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:49:06,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:49:06,633::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138322 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:49:10,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:49:10,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833223 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:49:16,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:49:16,692::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173239 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:49:20,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:49:20,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802848 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:49:26,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:49:26,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145511 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:49:30,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:49:30,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814222 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:49:36,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:49:36,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120337 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:49:40,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:49:40,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769156 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:49:46,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:49:46,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175777 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:49:50,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:49:50,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803231 s, 5.1 MB/s\n'; <rc> = 0
>Thread-13780::WARNING::2013-03-17 04:49:50,983::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:49:50,986::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 04:49:56,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:49:56,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140743 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:50:00,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:50:00,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159722 s, 256 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:50:00,988::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:50:00,989::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:50:01,676::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:50:01,684::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:50:01,686::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:50:06,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:50:06,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00298023 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:50:10,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:50:10,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00211663 s, 1.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:50:16,944::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:50:16,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788062 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:50:20,951::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 04:50:20,953::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 04:50:20,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:50:20,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00136898 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:50:20,998::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00136898 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 04:50:20,999::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 04:50:20,999::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 04:50:21,000::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 04:50:21,001::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:50:21,002::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:50:21,003::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:50:21,003::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:50:21,003::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:50:21,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:50:21,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:50:21,442::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:50:21,443::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 04:50:21,445::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 04:50:21,447::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 04:50:21,447::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 04:50:21,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:50:21,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:50:21,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:50:21,880::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109392 s, 374 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:50:26,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:50:27,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809056 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:50:31,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:50:31,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168358 s, 243 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:50:37,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:50:37,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00675381 s, 606 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:50:42,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:50:42,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153171 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:50:47,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:50:47,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168156 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:50:52,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:50:52,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826552 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:50:57,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:50:57,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150196 s, 2.7 MB/s\n'; <rc> = 0
>Thread-13781::WARNING::2013-03-17 04:51:01,712::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:51:01,716::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:51:02,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:51:02,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839476 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:51:07,177::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 04:51:07,179::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 04:51:07,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:51:07,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000871293 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:51:07,216::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000871293 s, 2.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 04:51:07,217::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 04:51:07,217::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 04:51:07,219::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 04:51:07,220::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:51:07,221::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:51:07,222::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:51:07,222::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:51:07,223::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:51:07,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:51:07,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:51:07,648::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:51:07,649::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 04:51:07,652::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 04:51:07,653::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 04:51:07,654::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 04:51:07,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:51:08,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:51:08,014::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:51:08,060::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187662 s, 218 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:51:11,718::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 04:51:11,719::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:51:11,720::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:51:12,200::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:51:12,214::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:51:12,216::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:51:12,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:51:12,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015615 s, 262 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:51:18,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:51:18,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000964298 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:51:22,412::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:51:22,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873168 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:51:28,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:51:28,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777576 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:51:32,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:51:32,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855733 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:51:38,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:51:38,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824268 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:51:42,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:51:42,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757022 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:51:48,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:51:48,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149842 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:51:52,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:51:52,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013356 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:51:58,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:51:58,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818729 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:52:02,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:52:02,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155096 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:52:08,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:52:08,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848364 s, 4.8 MB/s\n'; <rc> = 0
>Thread-13782::WARNING::2013-03-17 04:52:12,241::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:52:12,246::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:52:12,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:52:12,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0071172 s, 576 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:52:18,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:52:18,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836641 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:52:22,249::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:52:22,250::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:52:22,709::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:52:22,715::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:52:22,716::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:52:22,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:52:22,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127766 s, 321 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:52:28,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:52:28,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147767 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:52:32,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:52:32,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176459 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:52:38,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:52:38,401::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874523 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:52:43,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:52:43,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185872 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:52:48,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:52:48,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770746 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:52:53,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:52:53,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847677 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:52:58,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:52:58,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00299872 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:53:03,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:53:03,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821639 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:53:08,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:53:08,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153036 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:53:13,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:53:13,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851866 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:53:18,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:53:18,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148318 s, 2.8 MB/s\n'; <rc> = 0
>Thread-13783::WARNING::2013-03-17 04:53:22,742::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:53:22,746::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:53:23,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:53:23,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177967 s, 230 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:53:28,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:53:28,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817796 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:53:32,748::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:53:32,750::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:53:33,167::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:53:33,174::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:53:33,175::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:53:33,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:53:33,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00952789 s, 430 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:53:38,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:53:38,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000930554 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:53:43,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:53:43,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148268 s, 276 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:53:48,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:53:48,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815161 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:53:53,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:53:53,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101031 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:53:58,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:53:58,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864636 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:54:03,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:54:03,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151922 s, 270 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:54:08,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:54:08,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000910679 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:54:13,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:54:13,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0276622 s, 148 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:54:18,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:54:18,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00562603 s, 728 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:54:23,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:54:23,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017701 s, 231 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:54:28,857::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:54:28,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000938203 s, 4.4 MB/s\n'; <rc> = 0
>Thread-13784::WARNING::2013-03-17 04:54:33,200::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:54:33,203::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:54:33,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:54:33,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000908538 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:54:38,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:54:38,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747818 s, 5.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:54:43,206::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:54:43,207::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:54:43,612::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:54:43,618::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:54:43,620::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:54:44,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:54:44,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019712 s, 208 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:54:48,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:54:49,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014116 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:54:54,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:54:54,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844344 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:54:59,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:54:59,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000937252 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:55:04,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:55:04,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125903 s, 325 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:55:09,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:55:09,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140237 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:55:14,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:55:14,299::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000708771 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:55:19,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:55:19,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179157 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:55:24,337::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 04:55:24,339::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 04:55:24,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:55:24,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0106251 s, 193 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:55:24,382::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0106251 s, 193 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 04:55:24,383::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 04:55:24,384::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 04:55:24,387::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 04:55:24,388::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:55:24,389::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:55:24,389::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:55:24,390::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:55:24,390::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 04:55:24,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:55:24,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:55:24,864::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:55:24,869::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 04:55:24,871::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 04:55:24,873::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 04:55:24,873::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 04:55:24,875::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:55:25,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:55:25,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:55:25,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00460565 s, 889 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:55:29,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:55:29,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00605886 s, 676 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:55:35,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:55:35,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0200462 s, 204 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:55:39,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:55:39,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106662 s, 3.8 MB/s\n'; <rc> = 0
>Thread-13785::WARNING::2013-03-17 04:55:43,647::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:55:43,651::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:55:45,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:55:45,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129419 s, 316 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:55:49,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:55:49,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00191962 s, 2.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:55:53,653::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:55:53,655::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:55:54,053::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:55:54,060::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:55:54,061::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:55:55,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:55:55,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013139 s, 312 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:55:59,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:55:59,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0249148 s, 164 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:56:05,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:56:05,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898374 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:56:09,469::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 04:56:09,470::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 04:56:09,471::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:56:09,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000930401 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:56:09,510::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000930401 s, 2.2 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 04:56:09,511::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 04:56:09,511::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 04:56:09,513::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 04:56:09,514::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:56:09,514::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:56:09,515::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:56:09,515::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:56:09,515::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 04:56:09,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:56:09,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:56:09,939::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 04:56:09,939::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 04:56:09,942::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 04:56:09,943::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 04:56:09,944::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 04:56:09,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:56:10,311::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:56:10,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:56:10,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00541413 s, 757 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:56:15,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:56:15,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00780351 s, 525 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:56:20,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:56:20,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000984555 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:56:25,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:56:25,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120386 s, 340 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:56:30,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:56:30,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176949 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:56:35,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:56:35,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155159 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:56:40,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:56:40,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087887 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:56:45,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:56:46,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00724641 s, 565 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:56:50,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:56:50,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797637 s, 5.1 MB/s\n'; <rc> = 0
>Thread-13786::WARNING::2013-03-17 04:56:54,089::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:56:54,092::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:56:56,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:56:56,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878388 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:57:00,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:57:00,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814364 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:57:04,095::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 04:57:04,096::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:57:04,097::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:57:04,566::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:57:04,572::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:57:04,574::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:57:06,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:57:06,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117439 s, 349 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:57:10,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:57:10,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015909 s, 257 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:57:16,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:57:16,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125632 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:57:20,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:57:20,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000674625 s, 6.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:57:26,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:57:26,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000735107 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:57:30,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:57:30,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00845753 s, 484 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:57:36,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:57:36,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153667 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:57:40,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:57:40,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152219 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:57:46,445::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:57:46,483::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00392708 s, 1.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:57:50,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:57:50,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917234 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:57:56,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:57:56,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168473 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:58:00,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:58:00,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00180434 s, 2.3 MB/s\n'; <rc> = 0
>Thread-13787::WARNING::2013-03-17 04:58:04,599::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:58:04,603::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:58:06,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:58:06,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900969 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:58:10,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:58:10,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835629 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:58:14,605::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:58:14,607::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:58:15,027::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:58:15,034::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:58:15,035::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:58:16,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:58:16,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0098774 s, 415 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:58:20,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:58:20,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140115 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:58:26,760::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:58:26,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120408 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:58:30,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:58:30,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846984 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:58:36,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:58:36,881::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143848 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:58:41,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:58:41,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00186136 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:58:46,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:58:46,963::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178557 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:58:51,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:58:51,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174706 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:58:57,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:58:57,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817237 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:59:01,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:59:01,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148932 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:59:07,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:59:07,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120979 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:59:11,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:59:11,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154248 s, 2.7 MB/s\n'; <rc> = 0
>Thread-13788::WARNING::2013-03-17 04:59:15,078::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 04:59:15,084::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 04:59:17,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:59:17,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787039 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:59:21,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:59:21,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092803 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 04:59:25,097::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 04:59:25,099::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 04:59:25,502::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 04:59:25,509::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 04:59:25,510::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 04:59:27,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:59:27,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139583 s, 293 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:59:31,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:59:31,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794601 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:59:37,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:59:37,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859901 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:59:41,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:59:41,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013935 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:59:47,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:59:47,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770933 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 04:59:51,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 04:59:51,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00248126 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 04:59:57,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 04:59:57,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777604 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:00:01,345::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:00:01,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150255 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:00:07,528::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:00:07,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000977439 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:00:11,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:00:11,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016195 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:00:17,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:00:17,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761794 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:00:21,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:00:21,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00273041 s, 1.5 MB/s\n'; <rc> = 0
>Thread-13789::WARNING::2013-03-17 05:00:25,537::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:00:25,541::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:00:27,681::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 05:00:27,683::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 05:00:27,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:00:27,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000781595 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:00:27,738::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000781595 s, 2.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 05:00:27,738::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 05:00:27,738::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 05:00:27,740::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 05:00:27,741::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:00:27,742::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:00:27,742::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:00:27,743::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:00:27,743::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:00:27,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:00:28,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:00:28,166::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:00:28,167::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 05:00:28,169::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 05:00:28,170::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 05:00:28,171::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 05:00:28,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:00:28,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:00:28,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:00:28,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748332 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:00:31,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:00:31,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0235486 s, 174 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:00:35,543::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:00:35,545::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:00:36,013::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:00:36,050::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:00:36,051::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:00:38,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:00:38,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159673 s, 257 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:00:41,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:00:41,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178069 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:00:48,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:00:48,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089972 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:00:51,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:00:51,709::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148758 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:00:58,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:00:58,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794937 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:01:01,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:01:01,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101414 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:01:08,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:01:08,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832644 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:01:11,764::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 05:01:11,783::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 05:01:11,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:01:11,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00186686 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:01:11,815::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00186686 s, 1.1 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 05:01:11,815::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 05:01:11,816::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 05:01:11,818::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 05:01:11,819::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:01:11,820::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:01:11,821::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:01:11,821::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:01:11,822::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:01:11,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:01:12,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:01:12,256::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:01:12,257::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 05:01:12,259::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 05:01:12,260::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 05:01:12,260::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 05:01:12,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:01:12,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:01:12,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:01:12,801::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0233622 s, 175 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:01:18,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:01:18,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00970707 s, 422 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:01:22,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:01:22,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847824 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:01:28,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:01:29,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834959 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:01:32,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:01:32,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803048 s, 5.1 MB/s\n'; <rc> = 0
>Thread-13790::WARNING::2013-03-17 05:01:36,087::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:01:36,090::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:01:39,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:01:39,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834064 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:01:42,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:01:42,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808111 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:01:46,092::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:01:46,094::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:01:46,532::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:01:46,538::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:01:46,539::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:01:49,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:01:49,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00530965 s, 771 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:01:52,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:01:52,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110376 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:01:59,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:01:59,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00854157 s, 480 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:02:02,950::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:02:02,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00204195 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:02:09,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:02:09,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843001 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:02:13,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:02:13,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148618 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:02:19,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:02:19,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796473 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:02:23,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:02:23,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015528 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:02:29,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:02:29,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014583 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:02:33,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:02:33,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000724648 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:02:39,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:02:39,548::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000704613 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:02:43,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:02:43,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111565 s, 3.7 MB/s\n'; <rc> = 0
>Thread-13791::WARNING::2013-03-17 05:02:46,566::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:02:46,570::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:02:49,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:02:49,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171072 s, 239 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:02:53,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:02:53,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842031 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:02:56,576::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 05:02:56,578::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:02:56,579::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:02:57,003::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:02:57,010::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:02:57,012::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:02:59,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:02:59,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00976807 s, 419 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:03:03,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:03:03,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015566 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:03:09,778::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:03:09,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840981 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:03:13,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:03:13,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100432 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:03:19,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:03:19,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773349 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:03:23,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:03:23,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779322 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:03:29,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:03:29,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081571 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:03:33,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:03:33,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00300832 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:03:39,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:03:40,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00193223 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:03:43,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:03:43,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167862 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:03:50,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:03:50,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102697 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:03:53,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:03:53,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152318 s, 2.7 MB/s\n'; <rc> = 0
>Thread-13792::WARNING::2013-03-17 05:03:57,040::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:03:57,045::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:04:00,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:04:00,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811968 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:04:03,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:04:03,526::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757374 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:04:07,052::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:04:07,053::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:04:07,445::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:04:07,452::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:04:07,453::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:04:10,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:04:10,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174267 s, 235 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:04:13,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:04:13,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00183547 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:04:20,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:04:20,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000943206 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:04:23,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:04:23,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079113 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:04:30,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:04:30,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000914009 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:04:33,614::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:04:33,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834546 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:04:40,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:04:40,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00200883 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:04:43,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:04:43,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121659 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:04:50,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:04:50,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796908 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:04:53,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:04:53,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139571 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:05:00,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:05:00,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00268803 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:05:03,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:05:03,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832348 s, 4.9 MB/s\n'; <rc> = 0
>Thread-13793::WARNING::2013-03-17 05:05:07,480::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:05:07,486::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:05:10,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:05:10,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161541 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:05:13,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:05:13,837::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169898 s, 241 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:05:17,489::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:05:17,491::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:05:17,924::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:05:17,931::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:05:17,932::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:05:20,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:05:20,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014784 s, 277 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:05:23,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:05:23,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826008 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:05:30,839::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 05:05:30,841::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 05:05:30,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:05:30,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00091905 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:05:30,869::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00091905 s, 2.2 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 05:05:30,869::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 05:05:30,870::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 05:05:30,871::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 05:05:30,872::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:05:30,873::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:05:30,874::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:05:30,874::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:05:30,874::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:05:30,875::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:05:31,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:05:31,298::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:05:31,299::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 05:05:31,301::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 05:05:31,303::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 05:05:31,303::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 05:05:31,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:05:31,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:05:31,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:05:31,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00837264 s, 489 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:05:33,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:05:33,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133218 s, 307 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:05:41,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:05:41,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161086 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:05:43,933::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:05:43,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858751 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:05:51,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:05:51,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161029 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:05:53,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:05:53,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764539 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:06:01,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:06:02,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113361 s, 361 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:06:04,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:06:04,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00703319 s, 582 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:06:12,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:06:12,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177622 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:06:14,048::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 05:06:14,049::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 05:06:14,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:06:14,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00138309 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:06:14,091::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00138309 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 05:06:14,092::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 05:06:14,092::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 05:06:14,093::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 05:06:14,098::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:06:14,099::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:06:14,099::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:06:14,099::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:06:14,100::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:06:14,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:06:14,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:06:14,504::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:06:14,505::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 05:06:14,508::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 05:06:14,509::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 05:06:14,510::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 05:06:14,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:06:14,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:06:14,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:06:14,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172064 s, 238 kB/s\n'; <rc> = 0
>Thread-13794::WARNING::2013-03-17 05:06:17,964::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:06:17,968::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:06:22,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:06:22,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793658 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:06:24,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:06:24,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000998775 s, 4.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:06:27,970::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:06:27,971::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:06:28,388::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:06:28,395::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:06:28,396::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:06:32,305::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:06:32,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00825312 s, 496 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:06:34,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:06:35,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145963 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:06:42,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:06:42,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153619 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:06:45,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:06:45,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0220425 s, 186 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:06:52,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:06:52,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174564 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:06:55,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:06:55,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849496 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:07:02,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:07:02,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152037 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:07:05,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:07:05,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822664 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:07:12,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:07:12,667::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078837 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:07:15,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:07:15,191::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132168 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:07:22,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:07:22,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100428 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:07:25,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:07:25,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146209 s, 2.8 MB/s\n'; <rc> = 0
>Thread-13795::WARNING::2013-03-17 05:07:28,419::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:07:28,422::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:07:32,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:07:32,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100516 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:07:35,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:07:35,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838883 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:07:38,425::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:07:38,426::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:07:38,836::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:07:38,843::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:07:38,844::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:07:42,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:07:42,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146506 s, 280 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:07:45,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:07:45,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802844 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:07:52,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:07:52,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856134 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:07:55,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:07:55,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080982 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:08:03,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:08:03,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901339 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:08:05,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:08:05,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107298 s, 382 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:08:13,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:08:13,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000927861 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:08:15,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:08:15,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148855 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:08:23,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:08:23,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176368 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:08:25,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:08:25,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806223 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:08:33,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:08:33,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138245 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:08:35,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:08:35,561::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104594 s, 3.9 MB/s\n'; <rc> = 0
>Thread-13796::WARNING::2013-03-17 05:08:38,871::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:08:38,875::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:08:43,352::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:08:43,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103554 s, 396 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:08:45,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:08:45,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134521 s, 3.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:08:48,877::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 05:08:48,878::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:08:48,880::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:08:49,296::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:08:49,304::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:08:49,306::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:08:53,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:08:53,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134754 s, 304 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:08:55,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:08:55,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014256 s, 287 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:09:03,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:09:03,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770282 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:09:05,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:09:05,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00821037 s, 499 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:09:13,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:09:13,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00195425 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:09:15,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:09:15,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160665 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:09:23,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:09:23,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154797 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:09:25,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:09:25,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146519 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:09:33,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:09:33,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133187 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:09:35,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:09:35,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104061 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:09:43,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:09:43,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000922384 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:09:45,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:09:45,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105329 s, 3.9 MB/s\n'; <rc> = 0
>Thread-13797::WARNING::2013-03-17 05:09:49,338::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:09:49,344::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:09:53,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:09:53,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135326 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:09:55,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:09:55,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016389 s, 2.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:09:59,346::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:09:59,348::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:09:59,742::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:09:59,749::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:09:59,751::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:10:04,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:10:04,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0332613 s, 123 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:10:05,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:10:06,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00273734 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:10:14,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:10:14,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015553 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:10:16,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:10:16,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000720555 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:10:24,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:10:24,266::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116561 s, 351 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:10:26,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:10:26,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141332 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:10:34,306::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 05:10:34,307::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 05:10:34,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:10:34,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00144559 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:10:34,356::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00144559 s, 1.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 05:10:34,357::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 05:10:34,358::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 05:10:34,359::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 05:10:34,361::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:10:34,361::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:10:34,362::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:10:34,362::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:10:34,362::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:10:34,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:10:34,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:10:34,766::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:10:34,767::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 05:10:34,769::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 05:10:34,770::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 05:10:34,771::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 05:10:34,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:10:35,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:10:35,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:10:35,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0276418 s, 148 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:10:36,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:10:36,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00546283 s, 750 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:10:45,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:10:45,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842533 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:10:46,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:10:46,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118291 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:10:55,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:10:55,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000691474 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:10:56,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:10:56,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00183719 s, 2.2 MB/s\n'; <rc> = 0
>Thread-13798::WARNING::2013-03-17 05:10:59,775::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:10:59,782::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:11:05,596::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:11:05,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000944369 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:11:06,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:11:06,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100852 s, 4.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:11:09,785::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:11:09,786::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:11:10,203::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:11:10,210::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:11:10,212::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:11:15,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:11:15,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0078425 s, 522 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:11:16,420::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 05:11:16,422::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 05:11:16,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:11:16,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00174504 s, 1.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:11:16,490::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00174504 s, 1.2 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 05:11:16,492::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 05:11:16,492::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 05:11:16,493::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 05:11:16,494::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:11:16,495::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:11:16,495::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:11:16,496::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:11:16,497::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:11:16,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:11:16,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:11:16,920::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:11:16,922::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 05:11:16,925::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 05:11:16,926::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 05:11:16,927::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 05:11:16,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:11:17,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:11:17,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:11:17,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134423 s, 305 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:11:25,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:11:25,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157584 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:11:27,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:11:27,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763763 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:11:35,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:11:35,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00243177 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:11:37,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:11:37,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0210085 s, 195 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:11:45,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:11:45,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000938921 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:11:47,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:11:47,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150666 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:11:55,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:11:56,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000721854 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:11:57,537::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:11:57,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173332 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:12:06,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:12:06,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122181 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:12:07,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:12:07,622::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163568 s, 2.5 MB/s\n'; <rc> = 0
>Thread-13799::WARNING::2013-03-17 05:12:10,238::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:12:10,246::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:12:16,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:12:16,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796049 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:12:17,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:12:17,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891971 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:12:20,250::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:12:20,251::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:12:20,709::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:12:20,715::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:12:20,717::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:12:26,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:12:26,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139485 s, 294 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:12:27,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:12:27,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00627283 s, 653 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:12:36,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:12:36,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155854 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:12:37,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:12:37,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878892 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:12:46,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:12:46,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793396 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:12:47,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:12:47,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154125 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:12:56,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:12:56,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176276 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:12:57,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:12:57,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877039 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:13:06,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:13:06,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144708 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:13:07,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:13:07,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085292 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:13:16,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:13:16,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157904 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:13:17,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:13:17,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141406 s, 2.9 MB/s\n'; <rc> = 0
>Thread-13800::WARNING::2013-03-17 05:13:20,747::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:13:20,753::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:13:26,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:13:26,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800432 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:13:27,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:13:28,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079054 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:13:30,756::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:13:30,757::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:13:31,247::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:13:31,253::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:13:31,255::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:13:36,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:13:36,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0075552 s, 542 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:13:38,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:13:38,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0018535 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:13:46,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:13:46,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781516 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:13:48,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:13:48,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885699 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:13:56,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:13:57,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807233 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:13:58,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:13:58,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104483 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:14:07,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:14:07,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843043 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:14:08,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:14:08,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139879 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:14:17,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:14:17,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785448 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:14:18,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:14:18,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888776 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:14:27,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:14:27,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104796 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:14:28,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:14:28,266::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00190108 s, 2.2 MB/s\n'; <rc> = 0
>Thread-13801::WARNING::2013-03-17 05:14:31,282::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:14:31,286::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:14:37,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:14:37,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149936 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:14:38,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:14:38,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890458 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:14:41,288::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 05:14:41,290::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:14:41,292::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:14:41,779::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:14:41,786::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:14:41,787::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:14:47,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:14:47,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189301 s, 216 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:14:48,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:14:48,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00346857 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:14:57,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:14:57,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160337 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:14:58,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:14:58,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000735178 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:15:07,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:15:07,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083023 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:15:08,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:15:08,439::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178779 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:15:17,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:15:17,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000715048 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:15:18,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:15:18,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015244 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:15:27,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:15:27,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000713384 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:15:28,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:15:28,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155252 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:15:37,802::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 05:15:37,803::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 05:15:37,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:15:37,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0211814 s, 96.7 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:15:37,893::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0211814 s, 96.7 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 05:15:37,894::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 05:15:37,894::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 05:15:37,896::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 05:15:37,897::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:15:37,898::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:15:37,898::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:15:37,899::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:15:37,899::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:15:37,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:15:38,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:15:38,379::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:15:38,381::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 05:15:38,385::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 05:15:38,386::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 05:15:38,390::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 05:15:38,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:15:38,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:15:38,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109783 s, 373 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:15:38,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:15:38,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:15:38,853::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00825571 s, 496 kB/s\n'; <rc> = 0
>Thread-13802::WARNING::2013-03-17 05:15:41,815::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:15:41,818::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 05:15:48,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:15:48,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896692 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:15:48,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:15:48,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862925 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:15:51,823::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:15:51,825::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:15:52,368::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:15:52,375::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:15:52,377::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:15:58,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:15:58,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743078 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:15:58,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:15:59,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182334 s, 225 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:16:08,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:16:08,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898013 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:16:09,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:16:09,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155984 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:16:18,730::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 05:16:18,731::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 05:16:18,732::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:16:18,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00145286 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:16:18,771::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00145286 s, 1.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 05:16:18,772::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 05:16:18,772::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 05:16:18,774::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 05:16:18,775::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:16:18,775::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:16:18,776::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:16:18,776::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:16:18,776::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:16:18,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:16:19,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:16:19,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:16:19,235::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:16:19,240::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 05:16:19,254::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-20::DEBUG::2013-03-17 05:16:19,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134307 s, 305 kB/s\n'; <rc> = 0
>Thread-22::WARNING::2013-03-17 05:16:19,257::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 05:16:19,260::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 05:16:19,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:16:19,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:16:19,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:16:19,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127962 s, 320 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:16:29,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:16:29,398::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000903586 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:16:29,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:16:29,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157437 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:16:39,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:16:39,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014267 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:16:39,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:16:39,881::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864031 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:16:49,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:16:49,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0037776 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:16:49,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:16:49,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00289724 s, 1.4 MB/s\n'; <rc> = 0
>Thread-13803::WARNING::2013-03-17 05:16:52,405::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:16:52,410::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:16:59,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:16:59,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082777 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:16:59,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:16:59,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786938 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:17:02,412::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:17:02,413::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:17:02,896::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:17:02,903::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:17:02,904::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:17:09,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:17:09,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155527 s, 263 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:17:09,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:17:10,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011452 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:17:19,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:17:19,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000739742 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:17:20,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:17:20,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000969882 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:17:29,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:17:29,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000942674 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:17:30,048::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:17:30,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779255 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:17:39,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:17:40,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074306 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:17:40,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:17:40,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000942441 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:17:50,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:17:50,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148317 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:17:50,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:17:50,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00201854 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:18:00,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:18:00,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:18:00,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106503 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:18:00,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899427 s, 4.6 MB/s\n'; <rc> = 0
>Thread-13804::WARNING::2013-03-17 05:18:02,936::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:18:02,940::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 05:18:10,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:18:10,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104846 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:18:10,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:18:10,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769323 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:18:12,948::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:18:12,949::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:18:13,385::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:18:13,391::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:18:13,393::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:18:20,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:18:20,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786788 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:18:20,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:18:20,395::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849362 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:18:30,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:18:30,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139993 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:18:30,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:18:30,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154332 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:18:40,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:18:40,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131161 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:18:40,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:18:40,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145614 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:18:50,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:18:50,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803778 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:18:50,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:18:50,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157544 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:19:00,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:19:00,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775549 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:19:00,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:19:00,734::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144518 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:19:10,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:19:10,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000998577 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:19:10,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:19:10,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175153 s, 2.3 MB/s\n'; <rc> = 0
>Thread-13805::WARNING::2013-03-17 05:19:13,420::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:19:13,424::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 05:19:20,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:19:20,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170171 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:19:20,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:19:20,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100516 s, 4.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:19:23,426::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:19:23,428::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:19:23,876::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:19:23,883::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:19:23,884::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:19:30,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:19:30,627::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089619 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:19:30,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:19:30,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816235 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:19:40,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:19:40,681::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133244 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:19:41,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:19:41,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779659 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:19:50,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:19:50,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00319839 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:19:51,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:19:51,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152391 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:20:00,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:20:00,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010034 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:20:01,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:20:01,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146151 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:20:10,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:20:10,837::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00316297 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:20:11,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:20:11,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00434689 s, 942 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:20:20,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:20:20,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000621822 s, 6.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:20:21,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:20:21,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143931 s, 2.8 MB/s\n'; <rc> = 0
>Thread-13806::WARNING::2013-03-17 05:20:23,926::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:20:23,931::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 05:20:30,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:20:30,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902454 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:20:31,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:20:31,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00601094 s, 681 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:20:33,933::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 05:20:33,934::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:20:33,935::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:20:34,337::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:20:34,343::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:20:34,345::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:20:40,922::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:20:40,950::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835259 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:20:41,482::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 05:20:41,484::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 05:20:41,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:20:41,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00503248 s, 407 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:20:41,518::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00503248 s, 407 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 05:20:41,519::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 05:20:41,519::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 05:20:41,521::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 05:20:41,522::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:20:41,523::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:20:41,523::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:20:41,524::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:20:41,524::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:20:41,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:20:41,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:20:41,960::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:20:41,961::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 05:20:41,964::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 05:20:41,965::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 05:20:41,966::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 05:20:41,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:20:42,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:20:42,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:20:42,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00883056 s, 464 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:20:50,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:20:50,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886816 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:20:52,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:20:52,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827992 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:21:00,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:21:01,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913863 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:21:02,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:21:02,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167769 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:21:11,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:21:11,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135209 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:21:12,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:21:12,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167034 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:21:21,076::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 05:21:21,077::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 05:21:21,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:21:21,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00074237 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:21:21,109::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00074237 s, 2.8 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 05:21:21,109::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 05:21:21,109::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 05:21:21,111::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 05:21:21,113::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:21:21,114::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:21:21,114::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:21:21,115::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:21:21,115::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:21:21,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:21:21,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:21:21,535::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:21:21,536::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 05:21:21,538::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 05:21:21,540::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 05:21:21,540::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 05:21:21,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:21:21,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:21:21,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:21:21,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00872155 s, 470 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:21:22,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:21:22,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00561734 s, 729 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:21:31,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:21:31,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000978909 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:21:32,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:21:32,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000977559 s, 4.2 MB/s\n'; <rc> = 0
>Thread-13807::WARNING::2013-03-17 05:21:34,370::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:21:34,376::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 05:21:41,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:21:42,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888713 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:21:42,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:21:42,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912993 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:21:44,379::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:21:44,380::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:21:44,846::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:21:44,853::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:21:44,854::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:21:52,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:21:52,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850161 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:21:52,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:21:52,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859959 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:22:02,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:22:02,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172047 s, 238 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:22:02,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:22:03,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847528 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:22:12,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:22:12,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859629 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:22:13,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:22:13,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145328 s, 282 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:22:22,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:22:22,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146032 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:22:23,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:22:23,195::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147736 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:22:32,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:22:32,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140171 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:22:33,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:22:33,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784743 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:22:42,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:22:42,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000721727 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:22:43,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:22:43,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158962 s, 2.6 MB/s\n'; <rc> = 0
>Thread-13808::WARNING::2013-03-17 05:22:44,879::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:22:44,884::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 05:22:52,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:22:52,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0027275 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:22:53,560::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:22:53,591::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00284982 s, 1.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:22:54,887::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:22:54,888::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:22:55,294::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:22:55,301::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:22:55,302::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:23:02,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:23:02,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000964356 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:23:03,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:23:03,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792314 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:23:12,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:23:12,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167812 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:23:13,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:23:13,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154418 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:23:22,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:23:22,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154194 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:23:23,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:23:23,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100635 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:23:32,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:23:32,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773476 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:23:33,859::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:23:33,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142217 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:23:42,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:23:42,631::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757635 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:23:43,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:23:43,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752607 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:23:52,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:23:52,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737833 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:23:54,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:23:54,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150683 s, 2.7 MB/s\n'; <rc> = 0
>Thread-13809::WARNING::2013-03-17 05:23:55,335::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:23:55,338::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 05:24:02,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:24:02,712::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909633 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:24:04,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:24:04,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102985 s, 398 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:24:05,341::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:24:05,342::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:24:05,811::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:24:05,817::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:24:05,819::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:24:12,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:24:12,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162459 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:24:14,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:24:14,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000982205 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:24:22,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:24:22,805::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152086 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:24:24,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:24:24,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000935679 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:24:32,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:24:32,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088573 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:24:34,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:24:34,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109091 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:24:42,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:24:42,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0021876 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:24:44,471::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:24:44,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116957 s, 350 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:24:52,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:24:52,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167045 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:24:54,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:24:54,598::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000987593 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:25:02,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:25:02,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00379437 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:25:04,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:25:04,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00742423 s, 552 kB/s\n'; <rc> = 0
>Thread-13810::WARNING::2013-03-17 05:25:05,850::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:25:05,855::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 05:25:12,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:25:13,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849009 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:25:14,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:25:14,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837032 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:25:15,857::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:25:15,858::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:25:16,315::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:25:16,322::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:25:16,324::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:25:23,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:25:23,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765751 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:25:24,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:25:24,950::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00242643 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:25:33,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:25:33,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819064 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:25:34,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:25:35,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149271 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:25:43,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:25:43,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.021663 s, 189 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:25:45,073::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 05:25:45,075::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 05:25:45,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:25:45,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0207304 s, 98.8 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:25:45,124::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0207304 s, 98.8 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 05:25:45,124::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 05:25:45,124::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 05:25:45,126::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 05:25:45,127::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:25:45,128::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:25:45,128::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:25:45,129::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:25:45,129::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:25:45,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:25:45,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:25:45,622::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:25:45,623::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 05:25:45,625::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 05:25:45,626::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 05:25:45,627::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 05:25:45,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:25:45,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:25:46,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:25:46,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134098 s, 305 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:25:53,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:25:53,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160724 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:25:56,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:25:56,123::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150806 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:26:03,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:26:03,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206164 s, 199 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:26:06,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:26:06,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150165 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:26:13,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:26:13,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00440204 s, 930 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:26:16,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:26:16,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810127 s, 5.1 MB/s\n'; <rc> = 0
>Thread-13811::WARNING::2013-03-17 05:26:16,350::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:26:16,359::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 05:26:23,334::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 05:26:23,335::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 05:26:23,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:26:23,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00147614 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:26:23,369::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00147614 s, 1.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 05:26:23,370::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 05:26:23,370::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 05:26:23,372::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 05:26:23,373::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:26:23,374::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:26:23,374::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:26:23,374::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:26:23,375::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:26:23,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:26:23,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:26:23,780::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:26:23,780::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 05:26:23,783::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 05:26:23,784::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 05:26:23,784::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 05:26:23,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:26:24,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:26:24,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:26:24,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137167 s, 299 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:26:26,373::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 05:26:26,374::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:26:26,376::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:26:26,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:26:26,448::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144686 s, 283 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:26:26,885::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:26:26,891::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:26:26,893::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:26:34,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:26:34,236::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906769 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:26:36,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:26:36,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853168 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:26:44,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:26:44,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101208 s, 405 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:26:46,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:26:46,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00228573 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:26:54,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:26:54,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155568 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:26:56,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:26:56,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870537 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:27:04,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:27:04,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137565 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:27:06,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:27:06,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138418 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:27:14,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:27:14,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140265 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:27:16,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:27:16,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135074 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:27:24,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:27:24,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000994258 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:27:26,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-13812::WARNING::2013-03-17 05:27:26,938::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:27:26,964::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:27:26,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185074 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:27:34,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:27:34,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804382 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:27:36,969::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:27:36,970::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:27:37,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:27:37,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00971385 s, 422 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:27:37,457::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:27:37,462::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:27:37,463::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:27:44,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:27:44,548::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000693922 s, 5.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:27:47,117::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:27:47,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843688 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:27:54,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:27:54,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783367 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:27:57,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:27:57,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146227 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:28:04,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:28:04,633::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171419 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:28:07,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:28:07,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842388 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:28:14,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:28:14,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153222 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:28:17,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:28:17,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148335 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:28:24,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:28:24,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866784 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:28:27,415::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:28:27,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0279196 s, 147 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:28:34,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:28:34,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168432 s, 2.4 MB/s\n'; <rc> = 0
>Thread-13813::WARNING::2013-03-17 05:28:37,491::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:28:37,494::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:28:37,528::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:28:37,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746423 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:28:44,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:28:44,805::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858622 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:28:47,496::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:28:47,498::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:28:47,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:28:47,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107252 s, 382 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:28:47,948::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:28:47,954::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:28:47,955::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:28:54,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:28:54,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849249 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:28:57,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:28:57,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842462 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:29:04,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:29:04,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087314 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:29:07,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:29:07,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014217 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:29:14,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:29:14,912::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139246 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:29:17,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:29:17,881::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086765 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:29:24,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:29:24,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818399 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:29:27,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:29:27,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152948 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:29:34,950::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:29:35,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169184 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:29:38,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:29:38,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145994 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:29:45,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:29:45,046::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011403 s, 3.6 MB/s\n'; <rc> = 0
>Thread-13814::WARNING::2013-03-17 05:29:48,001::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:29:48,007::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:29:48,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:29:48,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152772 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:29:55,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:29:55,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000910777 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:29:58,010::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:29:58,012::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:29:58,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:29:58,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00981088 s, 417 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:29:58,462::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:29:58,469::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:29:58,471::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:30:05,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:30:05,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015184 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:30:08,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:30:08,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014282 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:30:15,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:30:15,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00192579 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:30:18,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:30:18,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00918074 s, 446 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:30:25,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:30:25,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010403 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:30:28,471::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:30:28,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000972819 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:30:35,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:30:35,238::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843295 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:30:38,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:30:38,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163532 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:30:45,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:30:45,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117326 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:30:48,631::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 05:30:48,633::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 05:30:48,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:30:48,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00913737 s, 224 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:30:48,683::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00913737 s, 224 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 05:30:48,683::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 05:30:48,684::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 05:30:48,685::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 05:30:48,686::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:30:48,687::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:30:48,687::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:30:48,687::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:30:48,688::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:30:48,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:30:49,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:30:49,092::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:30:49,093::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 05:30:49,096::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 05:30:49,097::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 05:30:49,098::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 05:30:49,099::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:30:49,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:30:49,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:30:49,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172758 s, 237 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:30:55,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:30:55,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860743 s, 4.8 MB/s\n'; <rc> = 0
>Thread-13815::WARNING::2013-03-17 05:30:58,506::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:30:58,511::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:30:59,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:30:59,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154814 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:31:05,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:31:05,366::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869006 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:31:08,514::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:31:08,516::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:31:08,945::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:31:08,952::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:31:08,953::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:31:09,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:31:09,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120311 s, 340 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:31:15,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:31:15,413::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139216 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:31:19,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:31:19,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155767 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:31:25,418::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 05:31:25,419::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 05:31:25,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:31:25,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000717504 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:31:25,449::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000717504 s, 2.9 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 05:31:25,449::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 05:31:25,449::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 05:31:25,451::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 05:31:25,452::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:31:25,453::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:31:25,453::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:31:25,453::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:31:25,454::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:31:25,455::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:31:25,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:31:25,889::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:31:25,890::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 05:31:25,892::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 05:31:25,894::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 05:31:25,894::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 05:31:25,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:31:26,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:31:26,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:31:26,309::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138249 s, 296 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:31:29,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:31:29,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203331 s, 201 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:31:36,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:31:36,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821402 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:31:39,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:31:39,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171369 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:31:46,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:31:46,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845673 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:31:50,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:31:50,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010817 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:31:56,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:31:56,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000889564 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:32:00,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:32:00,123::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013779 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:32:06,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:32:06,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010196 s, 4.0 MB/s\n'; <rc> = 0
>Thread-13816::WARNING::2013-03-17 05:32:08,982::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:32:08,989::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:32:10,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:32:10,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786708 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:32:16,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:32:16,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077558 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:32:18,993::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 05:32:18,994::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:32:18,996::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:32:19,412::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:32:19,419::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:32:19,421::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:32:20,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:32:20,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00708145 s, 578 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:32:26,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:32:26,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091969 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:32:30,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:32:30,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814628 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:32:36,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:32:36,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845444 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:32:40,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:32:40,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00970098 s, 422 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:32:46,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:32:46,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137705 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:32:50,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:32:50,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833976 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:32:56,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:32:56,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151173 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:33:00,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:33:00,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00564263 s, 726 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:33:06,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:33:06,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000735126 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:33:10,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:33:10,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901548 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:33:16,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:33:16,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140282 s, 2.9 MB/s\n'; <rc> = 0
>Thread-13817::WARNING::2013-03-17 05:33:19,451::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:33:19,454::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:33:20,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:33:20,742::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00949682 s, 431 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:33:26,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:33:26,791::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846944 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:33:29,456::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:33:29,457::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:33:29,853::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:33:29,859::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:33:29,861::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:33:30,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:33:30,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128434 s, 319 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:33:36,796::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:33:36,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101491 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:33:40,871::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:33:40,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00855046 s, 479 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:33:46,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:33:46,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741302 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:33:50,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:33:50,992::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131629 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:33:56,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:33:56,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132465 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:34:01,033::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:34:01,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165122 s, 248 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:34:06,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:34:06,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751389 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:34:11,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:34:11,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892215 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:34:16,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:34:17,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155001 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:34:21,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:34:21,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168398 s, 243 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:34:27,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:34:27,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159035 s, 2.6 MB/s\n'; <rc> = 0
>Thread-13818::WARNING::2013-03-17 05:34:29,884::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:34:29,887::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:34:31,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:34:31,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822982 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:34:37,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:34:37,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010696 s, 3.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:34:39,889::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:34:39,891::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:34:40,304::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:34:40,310::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:34:40,312::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:34:41,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:34:41,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142009 s, 288 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:34:47,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:34:47,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923918 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:34:51,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:34:51,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879025 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:34:57,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:34:57,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155016 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:35:01,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:35:01,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130253 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:35:07,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:35:07,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000722988 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:35:11,605::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:35:11,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000991467 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:35:17,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:35:17,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00220653 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:35:21,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:35:21,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152942 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:35:27,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:35:27,392::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769801 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:35:31,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:35:31,793::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000729572 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:35:37,396::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:35:37,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131751 s, 3.1 MB/s\n'; <rc> = 0
>Thread-13819::WARNING::2013-03-17 05:35:40,343::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:35:40,352::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:35:41,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:35:41,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157259 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:35:47,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:35:47,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000726403 s, 5.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:35:50,356::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:35:50,357::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:35:50,773::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:35:50,780::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:35:50,781::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:35:51,917::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 05:35:51,920::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 05:35:51,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:35:51,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0203322 s, 101 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:35:51,972::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0203322 s, 101 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 05:35:51,973::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 05:35:51,973::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 05:35:51,975::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 05:35:51,977::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:35:51,977::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:35:51,978::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:35:51,978::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:35:51,979::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:35:51,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:35:52,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:35:52,336::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:35:52,337::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 05:35:52,340::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 05:35:52,341::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 05:35:52,341::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 05:35:52,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:35:52,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:35:52,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:35:52,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119296 s, 343 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:35:57,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:35:57,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145764 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:36:02,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:36:02,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881367 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:36:07,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:36:07,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00217093 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:36:12,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:36:12,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816067 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:36:17,606::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:36:17,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126398 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:36:22,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:36:22,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144396 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:36:27,670::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 05:36:27,671::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 05:36:27,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:36:27,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00300268 s, 682 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:36:27,705::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00300268 s, 682 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 05:36:27,706::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 05:36:27,706::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 05:36:27,708::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 05:36:27,709::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:36:27,709::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:36:27,710::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:36:27,710::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:36:27,711::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:36:27,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:36:28,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:36:28,154::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:36:28,155::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 05:36:28,157::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 05:36:28,159::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 05:36:28,159::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 05:36:28,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:36:28,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:36:28,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:36:28,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00749319 s, 547 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:36:32,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:36:33,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108732 s, 377 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:36:38,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:36:38,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143209 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:36:43,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:36:43,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769196 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:36:48,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:36:48,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131871 s, 3.1 MB/s\n'; <rc> = 0
>Thread-13820::WARNING::2013-03-17 05:36:50,805::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:36:50,809::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:36:53,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:36:53,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000950199 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:36:58,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:36:58,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836417 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:37:00,811::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:37:00,812::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:37:01,240::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:37:01,247::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:37:01,248::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:37:03,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:37:03,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103025 s, 398 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:37:08,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:37:08,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813086 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:37:13,324::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:37:13,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141822 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:37:18,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:37:18,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872091 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:37:23,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:37:23,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865523 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:37:28,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:37:28,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799556 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:37:33,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:37:33,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00224665 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:37:38,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:37:38,880::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00584575 s, 701 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:37:43,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:37:43,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00227427 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:37:48,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:37:48,928::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151877 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:37:53,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:37:53,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824387 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:37:58,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:37:58,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126074 s, 3.2 MB/s\n'; <rc> = 0
>Thread-13821::WARNING::2013-03-17 05:38:01,281::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:38:01,286::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:38:03,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:38:03,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833216 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:38:08,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:38:09,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000720503 s, 5.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:38:11,294::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 05:38:11,295::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:38:11,296::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:38:11,732::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:38:11,739::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:38:11,740::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:38:13,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:38:13,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015698 s, 261 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:38:19,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:38:19,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898215 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:38:23,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:38:23,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758172 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:38:29,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:38:29,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115655 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:38:33,922::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:38:33,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135881 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:38:39,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:38:39,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154095 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:38:44,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:38:44,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124527 s, 329 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:38:49,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:38:49,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864079 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:38:54,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:38:54,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163733 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:38:59,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:38:59,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832383 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:39:04,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:39:04,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153308 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:39:09,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:39:09,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145564 s, 2.8 MB/s\n'; <rc> = 0
>Thread-13822::WARNING::2013-03-17 05:39:11,767::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:39:11,772::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:39:14,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:39:14,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119703 s, 342 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:39:19,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:39:19,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768674 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:39:21,775::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:39:21,776::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:39:22,202::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:39:22,208::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:39:22,210::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:39:24,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:39:24,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0251724 s, 163 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:39:29,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:39:29,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834027 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:39:34,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:39:34,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851406 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:39:39,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:39:39,398::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00372052 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:39:44,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:39:44,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000705881 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:39:49,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:39:49,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161537 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:39:54,612::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:39:54,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842446 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:39:59,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:39:59,484::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887216 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:40:04,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:40:04,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882713 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:40:09,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:40:09,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874032 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:40:14,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:40:14,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001539 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:40:19,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:40:19,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173316 s, 2.4 MB/s\n'; <rc> = 0
>Thread-13823::WARNING::2013-03-17 05:40:22,234::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:40:22,241::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:40:24,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:40:24,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848597 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:40:29,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:40:29,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137816 s, 3.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:40:32,245::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:40:32,247::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:40:32,666::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:40:32,673::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:40:32,674::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:40:34,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:40:34,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017881 s, 229 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:40:39,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:40:39,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163334 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:40:44,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:40:45,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787747 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:40:49,681::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:40:49,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148978 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:40:55,053::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 05:40:55,054::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 05:40:55,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:40:55,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0148886 s, 138 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:40:55,109::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0148886 s, 138 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 05:40:55,110::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 05:40:55,110::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 05:40:55,112::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 05:40:55,113::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:40:55,114::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:40:55,114::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:40:55,114::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:40:55,116::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:40:55,117::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:40:55,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:40:55,575::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:40:55,576::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 05:40:55,579::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 05:40:55,580::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 05:40:55,580::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 05:40:55,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:40:55,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:40:55,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:40:56,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111859 s, 366 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:40:59,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:40:59,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142198 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:41:06,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:41:06,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835224 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:41:09,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:41:09,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145775 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:41:16,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:41:16,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123369 s, 332 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:41:19,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:41:19,866::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135371 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:41:26,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:41:26,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144101 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:41:29,870::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 05:41:29,872::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 05:41:29,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:41:29,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000799859 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:41:29,926::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000799859 s, 2.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 05:41:29,927::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 05:41:29,927::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 05:41:29,931::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 05:41:29,932::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:41:29,933::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:41:29,934::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:41:29,935::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:41:29,935::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:41:29,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:41:30,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:41:30,366::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:41:30,367::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 05:41:30,370::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 05:41:30,373::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 05:41:30,374::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 05:41:30,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:41:30,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:41:30,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:41:30,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0060171 s, 681 kB/s\n'; <rc> = 0
>Thread-13824::WARNING::2013-03-17 05:41:32,713::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:41:32,716::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:41:36,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:41:36,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153357 s, 267 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:41:40,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:41:40,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136939 s, 3.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:41:42,729::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:41:42,730::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:41:43,177::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:41:43,184::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:41:43,185::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:41:46,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:41:46,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130379 s, 314 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:41:50,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:41:50,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923951 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:41:56,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:41:56,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00903078 s, 454 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:42:00,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:42:00,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895293 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:42:06,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:42:06,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000987127 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:42:10,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:42:11,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871894 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:42:16,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:42:16,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172385 s, 238 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:42:21,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:42:21,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144457 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:42:26,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:42:26,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137196 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:42:31,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:42:31,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092836 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:42:36,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:42:36,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014958 s, 274 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:42:41,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:42:41,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091422 s, 4.5 MB/s\n'; <rc> = 0
>Thread-13825::WARNING::2013-03-17 05:42:43,213::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:42:43,217::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:42:46,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:42:46,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751361 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:42:51,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:42:51,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895042 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:42:53,220::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:42:53,222::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:42:53,636::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:42:53,644::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:42:53,646::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:42:56,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:42:57,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0086954 s, 471 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:43:01,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:43:01,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854302 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:43:07,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:43:07,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153711 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:43:11,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:43:11,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853719 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:43:17,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:43:17,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00903552 s, 453 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:43:21,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:43:21,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859443 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:43:27,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:43:27,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869457 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:43:31,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:43:31,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911801 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:43:37,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:43:37,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00761989 s, 538 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:43:41,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:43:41,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159498 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:43:47,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:43:47,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175059 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:43:51,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:43:51,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132444 s, 3.1 MB/s\n'; <rc> = 0
>Thread-13826::WARNING::2013-03-17 05:43:53,687::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:43:53,690::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:43:57,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:43:57,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00806396 s, 508 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:44:01,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:44:01,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766042 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:44:03,692::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 05:44:03,693::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:44:03,694::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:44:04,147::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:44:04,153::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:44:04,155::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:44:07,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:44:07,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110826 s, 370 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:44:11,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:44:11,517::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000914846 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:44:17,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:44:17,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00873678 s, 469 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:44:21,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:44:21,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00230662 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:44:27,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:44:27,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826349 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:44:31,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:44:31,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840414 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:44:37,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:44:37,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00692549 s, 591 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:44:41,605::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:44:41,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139766 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:44:47,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:44:47,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142336 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:44:51,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:44:51,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00258957 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:44:57,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:44:58,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139278 s, 294 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:45:01,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:45:01,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123273 s, 3.3 MB/s\n'; <rc> = 0
>Thread-13827::WARNING::2013-03-17 05:45:04,182::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:45:04,186::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:45:08,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:45:08,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796364 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:45:11,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:45:11,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000910132 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:45:14,192::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:45:14,193::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:45:14,669::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:45:14,676::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:45:14,678::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:45:18,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:45:18,163::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01101 s, 372 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:45:21,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:45:21,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140669 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:45:28,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:45:28,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000989705 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:45:31,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:45:31,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865472 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:45:38,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:45:38,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741299 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:45:41,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:45:41,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766816 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:45:48,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:45:48,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000703488 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:45:51,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:45:51,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799238 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:45:58,409::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 05:45:58,410::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 05:45:58,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:45:58,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0150657 s, 136 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:45:58,471::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0150657 s, 136 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 05:45:58,471::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 05:45:58,472::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 05:45:58,473::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 05:45:58,474::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:45:58,475::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:45:58,475::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:45:58,476::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:45:58,476::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:45:58,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:45:58,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:45:58,919::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:45:58,920::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 05:45:58,923::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 05:45:58,924::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 05:45:58,924::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 05:45:58,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:45:59,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:45:59,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:45:59,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00663423 s, 617 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:46:01,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:46:01,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161952 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:46:09,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:46:09,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799275 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:46:12,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:46:12,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122657 s, 3.3 MB/s\n'; <rc> = 0
>Thread-13828::WARNING::2013-03-17 05:46:14,709::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:46:14,716::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:46:19,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:46:19,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00331847 s, 1.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:46:22,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:46:22,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792283 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:46:24,721::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:46:24,722::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:46:25,147::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:46:25,154::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:46:25,155::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:46:29,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:46:29,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169288 s, 242 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:46:32,079::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 05:46:32,080::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 05:46:32,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:46:32,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000750913 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:46:32,109::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000750913 s, 2.7 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 05:46:32,109::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 05:46:32,109::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 05:46:32,111::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 05:46:32,112::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:46:32,113::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:46:32,113::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:46:32,113::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:46:32,114::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:46:32,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:46:32,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:46:32,542::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:46:32,543::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 05:46:32,546::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 05:46:32,547::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 05:46:32,547::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 05:46:32,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:46:32,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:46:32,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:46:32,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180838 s, 227 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:46:39,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:46:39,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00936677 s, 437 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:46:42,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:46:43,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157131 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:46:49,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:46:49,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779932 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:46:53,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:46:53,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160781 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:46:59,871::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:46:59,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858385 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:47:03,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:47:03,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000687615 s, 6.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:47:09,941::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:47:09,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151826 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:47:13,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:47:13,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150526 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:47:20,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:47:20,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174725 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:47:23,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:47:23,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014774 s, 2.8 MB/s\n'; <rc> = 0
>Thread-13829::WARNING::2013-03-17 05:47:25,185::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:47:25,188::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:47:30,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:47:30,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835138 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:47:33,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:47:33,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000750294 s, 5.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:47:35,201::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:47:35,202::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:47:35,630::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:47:35,638::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:47:35,640::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:47:40,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:47:40,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114844 s, 357 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:47:43,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:47:43,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120233 s, 341 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:47:50,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:47:50,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865925 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:47:53,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:47:53,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148195 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:48:00,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:48:00,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151167 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:48:03,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:48:03,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761154 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:48:10,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:48:10,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161592 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:48:13,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:48:13,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136232 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:48:20,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:48:20,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166787 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:48:23,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:48:23,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132931 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:48:30,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:48:30,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013984 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:48:33,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:48:33,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802795 s, 5.1 MB/s\n'; <rc> = 0
>Thread-13830::WARNING::2013-03-17 05:48:35,667::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:48:35,671::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:48:40,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:48:40,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167695 s, 244 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:48:43,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:48:43,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895658 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:48:45,673::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:48:45,726::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:48:46,159::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:48:46,166::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:48:46,168::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:48:50,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:48:50,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185922 s, 220 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:48:53,532::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:48:53,561::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894752 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:49:00,857::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:49:00,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186932 s, 219 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:49:03,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:49:03,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146144 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:49:10,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:49:11,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102309 s, 400 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:49:13,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:49:13,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872067 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:49:21,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:49:21,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149251 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:49:23,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:49:23,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900045 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:49:31,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:49:31,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197285 s, 208 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:49:33,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:49:33,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00834483 s, 491 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:49:41,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:49:41,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110166 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:49:43,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:49:43,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119768 s, 3.4 MB/s\n'; <rc> = 0
>Thread-13831::WARNING::2013-03-17 05:49:46,191::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:49:46,194::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:49:51,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:49:51,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0474857 s, 86.3 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:49:53,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:49:53,843::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810502 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:49:56,196::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 05:49:56,197::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:49:56,198::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:49:56,624::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:49:56,631::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:49:56,633::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:50:01,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:50:01,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00916009 s, 447 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:50:03,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:50:03,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854987 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:50:11,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:50:11,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156686 s, 261 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:50:13,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:50:13,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016313 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:50:21,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:50:21,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885759 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:50:23,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:50:23,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000690796 s, 5.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:50:31,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:50:31,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161033 s, 254 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:50:33,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:50:34,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000957583 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:50:41,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:50:41,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146612 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:50:44,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:50:44,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145491 s, 282 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:50:51,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:50:51,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168532 s, 243 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:50:54,079::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:50:54,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138477 s, 3.0 MB/s\n'; <rc> = 0
>Thread-13832::WARNING::2013-03-17 05:50:56,663::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:50:56,667::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:51:02,010::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 05:51:02,012::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 05:51:02,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:51:02,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000863086 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:51:02,044::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000863086 s, 2.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 05:51:02,044::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 05:51:02,045::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 05:51:02,047::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 05:51:02,048::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:51:02,049::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:51:02,050::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:51:02,050::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:51:02,051::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:51:02,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:51:02,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:51:02,446::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:51:02,447::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 05:51:02,450::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 05:51:02,451::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 05:51:02,451::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 05:51:02,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:51:02,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:51:02,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:51:02,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165888 s, 247 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:51:04,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:51:04,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104668 s, 391 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:51:06,680::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:51:06,681::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:51:07,113::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:51:07,120::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:51:07,121::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:51:12,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:51:12,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151079 s, 271 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:51:14,178::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:51:14,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00263206 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:51:22,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:51:23,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010212 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:51:24,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:51:24,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122597 s, 334 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:51:33,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:51:33,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151621 s, 270 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:51:34,263::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 05:51:34,265::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 05:51:34,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:51:34,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00156512 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:51:34,308::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00156512 s, 1.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 05:51:34,309::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 05:51:34,309::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 05:51:34,311::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 05:51:34,313::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:51:34,313::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:51:34,314::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:51:34,314::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:51:34,314::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:51:34,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:51:34,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:51:34,750::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:51:34,751::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 05:51:34,753::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 05:51:34,754::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 05:51:34,755::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 05:51:34,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:51:35,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:51:35,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:51:35,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176155 s, 233 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:51:43,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:51:43,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013226 s, 310 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:51:45,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:51:45,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132148 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:51:53,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:51:53,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123881 s, 331 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:51:55,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:51:55,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738797 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:52:03,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:52:03,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192664 s, 213 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:52:05,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:52:05,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145656 s, 2.8 MB/s\n'; <rc> = 0
>Thread-13833::WARNING::2013-03-17 05:52:07,150::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:52:07,157::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:52:13,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:52:13,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116049 s, 353 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:52:15,305::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:52:15,333::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831214 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:52:17,161::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:52:17,162::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:52:17,586::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:52:17,593::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:52:17,594::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:52:23,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:52:23,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174657 s, 235 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:52:25,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:52:25,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137199 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:52:33,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:52:33,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186647 s, 219 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:52:35,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:52:35,413::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860719 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:52:43,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:52:43,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151617 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:52:45,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:52:45,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.021771 s, 188 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:52:53,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:52:53,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144685 s, 283 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:52:55,530::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:52:55,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00275179 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:53:03,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:53:03,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0229699 s, 178 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:53:05,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:53:05,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173136 s, 237 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:53:13,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:53:14,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013547 s, 302 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:53:15,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:53:15,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00400013 s, 1.0 MB/s\n'; <rc> = 0
>Thread-13834::WARNING::2013-03-17 05:53:17,620::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:53:17,624::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:53:24,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:53:24,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110454 s, 371 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:53:25,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:53:25,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802271 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:53:27,626::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:53:27,628::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:53:28,061::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:53:28,068::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:53:28,069::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:53:34,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:53:34,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0209654 s, 195 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:53:35,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:53:35,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000987005 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:53:44,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:53:44,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124991 s, 328 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:53:45,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:53:45,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000715573 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:53:54,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:53:54,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011946 s, 343 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:53:55,950::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:53:55,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820301 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:54:04,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:54:04,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013822 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:54:05,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:54:06,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176412 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:54:14,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:54:14,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150085 s, 273 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:54:16,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:54:16,081::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139106 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:54:24,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:54:24,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001664 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:54:26,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:54:26,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130083 s, 3.1 MB/s\n'; <rc> = 0
>Thread-13835::WARNING::2013-03-17 05:54:28,096::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:54:28,100::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:54:34,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:54:34,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131155 s, 312 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:54:36,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:54:36,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855351 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:54:38,102::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:54:38,103::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:54:38,542::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:54:38,549::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:54:38,550::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:54:44,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:54:44,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148526 s, 276 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:54:46,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:54:46,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000929793 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:54:54,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:54:54,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183953 s, 223 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:54:56,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:54:56,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765034 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:55:04,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:55:05,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0437748 s, 93.6 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:55:06,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:55:06,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811259 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:55:15,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:55:15,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100771 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:55:16,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:55:16,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768289 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:55:25,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:55:25,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741639 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:55:26,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:55:26,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144972 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:55:35,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:55:35,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793139 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:55:36,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:55:36,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906473 s, 4.5 MB/s\n'; <rc> = 0
>Thread-13836::WARNING::2013-03-17 05:55:38,581::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:55:38,586::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:55:45,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:55:45,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080978 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:55:46,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:55:46,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847545 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:55:48,592::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 05:55:48,593::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:55:48,595::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:55:49,066::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:55:49,073::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:55:49,074::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:55:55,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:55:55,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154745 s, 265 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:55:56,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:55:56,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925881 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:56:05,642::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 05:56:05,643::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 05:56:05,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:56:05,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00115446 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:56:05,697::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00115446 s, 1.8 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 05:56:05,697::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 05:56:05,698::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 05:56:05,699::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 05:56:05,700::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:56:05,701::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:56:05,701::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:56:05,702::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:56:05,702::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 05:56:05,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:56:06,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:56:06,142::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:56:06,143::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 05:56:06,145::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 05:56:06,146::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 05:56:06,147::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 05:56:06,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:56:06,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:56:06,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0232332 s, 176 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:56:06,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:56:06,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:56:06,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153593 s, 267 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:56:16,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:56:16,622::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154806 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:56:16,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:56:16,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150352 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:56:26,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:56:26,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788577 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:56:26,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:56:26,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147726 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:56:36,661::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 05:56:36,662::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 05:56:36,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:56:36,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00135835 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:56:36,703::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00135835 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 05:56:36,704::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 05:56:36,704::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 05:56:36,706::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 05:56:36,710::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:56:36,711::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:56:36,711::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:56:36,712::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:56:36,712::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 05:56:36,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:56:36,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:56:36,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148596 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:56:37,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:56:37,149::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 05:56:37,150::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 05:56:37,152::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 05:56:37,153::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 05:56:37,154::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 05:56:37,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:56:37,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:56:37,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:56:37,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00454029 s, 902 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:56:46,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:56:47,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00846034 s, 484 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:56:47,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:56:47,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808488 s, 5.1 MB/s\n'; <rc> = 0
>Thread-13837::WARNING::2013-03-17 05:56:49,110::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:56:49,113::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:56:57,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:56:57,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121804 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:56:57,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:56:57,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101209 s, 4.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:56:59,115::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:56:59,116::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:56:59,567::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:56:59,574::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:56:59,575::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:57:07,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:57:07,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153714 s, 266 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:57:07,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:57:07,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113256 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:57:17,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:57:17,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913968 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:57:17,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:57:17,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916158 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:57:27,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:57:27,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891896 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:57:27,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:57:27,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855748 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:57:37,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:57:37,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140511 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:57:37,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:57:37,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155747 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:57:47,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:57:47,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845003 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:57:47,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:57:47,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122293 s, 335 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:57:57,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:57:57,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131816 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:57:58,033::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:57:58,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120129 s, 3.4 MB/s\n'; <rc> = 0
>Thread-13838::WARNING::2013-03-17 05:57:59,603::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:57:59,607::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:58:07,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:58:07,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000983022 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:58:08,171::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:58:08,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151824 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:58:09,610::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:58:09,612::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:58:10,048::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:58:10,054::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:58:10,056::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:58:17,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:58:17,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185547 s, 221 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:58:18,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:58:18,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850922 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:58:27,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:58:27,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767713 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:58:28,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:58:28,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897372 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:58:37,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:58:37,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139383 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:58:38,305::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:58:38,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00182955 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:58:47,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:58:48,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139877 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:58:48,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:58:48,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014314 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:58:58,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:58:58,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015267 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:58:58,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:58:58,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759739 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:59:08,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:59:08,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197665 s, 207 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:59:08,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:59:08,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146977 s, 2.8 MB/s\n'; <rc> = 0
>Thread-13839::WARNING::2013-03-17 05:59:10,088::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 05:59:10,092::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 05:59:18,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:59:18,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790443 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:59:18,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:59:18,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870417 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 05:59:20,095::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 05:59:20,096::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 05:59:20,577::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 05:59:20,583::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 05:59:20,585::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 05:59:28,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:59:28,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183406 s, 223 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:59:28,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:59:28,552::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016008 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:59:38,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:59:38,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00209211 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:59:38,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:59:38,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000946793 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:59:48,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:59:48,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112247 s, 365 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:59:48,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:59:48,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00090615 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 05:59:58,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 05:59:58,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148828 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 05:59:58,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 05:59:58,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162934 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:00:08,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:00:08,778::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:00:08,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876014 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:00:08,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108371 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:00:18,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:00:18,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743671 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:00:18,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:00:18,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000969324 s, 4.2 MB/s\n'; <rc> = 0
>Thread-13840::WARNING::2013-03-17 06:00:20,614::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:00:20,619::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:00:28,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:00:28,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896872 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:00:28,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:00:28,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863973 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:00:30,630::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:00:30,631::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:00:31,041::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:00:31,048::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:00:31,050::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:00:38,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:00:38,963::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014912 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:00:39,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:00:39,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015373 s, 266 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:00:48,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:00:49,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0028841 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:00:49,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:00:49,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186443 s, 220 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:00:59,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:00:59,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079563 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:00:59,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:00:59,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187188 s, 219 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:01:09,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:01:09,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176777 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:01:09,298::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 06:01:09,300::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 06:01:09,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:01:09,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.01128 s, 182 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:01:09,344::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.01128 s, 182 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 06:01:09,344::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 06:01:09,344::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 06:01:09,350::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 06:01:09,351::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:01:09,352::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:01:09,353::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:01:09,353::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:01:09,353::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:01:09,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:01:09,777::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:01:09,781::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:01:09,783::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 06:01:09,785::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 06:01:09,786::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 06:01:09,787::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 06:01:09,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:01:10,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:01:10,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:01:10,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00351605 s, 1.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:01:19,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:01:19,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817897 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:01:20,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:01:20,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792926 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:01:29,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:01:29,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00318323 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:01:30,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:01:30,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133025 s, 3.1 MB/s\n'; <rc> = 0
>Thread-13841::WARNING::2013-03-17 06:01:31,076::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:01:31,081::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:01:39,170::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 06:01:39,171::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 06:01:39,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:01:39,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000893072 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:01:39,200::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000893072 s, 2.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 06:01:39,200::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 06:01:39,200::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 06:01:39,202::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 06:01:39,203::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:01:39,204::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:01:39,204::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:01:39,204::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:01:39,205::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:01:39,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:01:39,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:01:39,639::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:01:39,640::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 06:01:39,642::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 06:01:39,644::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 06:01:39,644::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 06:01:39,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:01:40,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:01:40,070::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:01:40,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00274731 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:01:40,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:01:40,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00754421 s, 543 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:01:41,082::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 06:01:41,083::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:01:41,084::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:01:41,514::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:01:41,521::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:01:41,523::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:01:50,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:01:50,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803062 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:01:50,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:01:50,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883208 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:02:00,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:02:00,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769775 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:02:00,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:02:01,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841227 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:02:10,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:02:10,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153069 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:02:11,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:02:11,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856288 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:02:20,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:02:20,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204588 s, 200 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:02:21,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:02:21,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769744 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:02:30,271::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:02:30,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794783 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:02:31,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:02:31,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143496 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:02:40,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:02:40,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110849 s, 370 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:02:41,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:02:41,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123325 s, 3.3 MB/s\n'; <rc> = 0
>Thread-13842::WARNING::2013-03-17 06:02:41,550::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:02:41,560::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:02:50,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:02:50,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800198 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:02:51,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:02:51,755::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:02:51,828::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:02:51,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191062 s, 214 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:02:52,346::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:02:52,351::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:02:52,355::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:03:00,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:03:00,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174148 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:03:01,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:03:01,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783447 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:03:10,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:03:10,709::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872306 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:03:12,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:03:12,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00481727 s, 850 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:03:20,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:03:20,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872034 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:03:22,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:03:22,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00207569 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:03:30,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:03:30,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147129 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:03:32,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:03:32,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149072 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:03:40,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:03:40,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820727 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:03:42,260::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:03:42,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171108 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:03:50,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:03:51,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825348 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:03:52,345::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-13843::WARNING::2013-03-17 06:03:52,397::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-20::DEBUG::2013-03-17 06:03:52,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787753 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::ERROR::2013-03-17 06:03:52,416::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:04:01,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:04:01,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100519 s, 4.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:04:02,433::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:04:02,435::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:04:02,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:04:02,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100354 s, 408 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:04:02,871::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:04:02,877::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:04:02,878::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:04:11,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:04:11,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832072 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:04:12,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:04:12,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156888 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:04:21,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:04:21,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880421 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:04:22,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:04:22,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803813 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:04:31,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:04:31,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895252 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:04:32,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:04:32,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105901 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:04:41,221::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:04:41,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00243613 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:04:42,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:04:42,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131687 s, 311 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:04:51,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:04:51,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00186952 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:04:52,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:04:52,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853719 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:05:01,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:05:01,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895993 s, 4.6 MB/s\n'; <rc> = 0
>Thread-13844::WARNING::2013-03-17 06:05:02,914::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:05:02,918::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 06:05:02,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:05:02,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091603 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:05:11,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:05:11,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917287 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:05:12,921::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:05:12,922::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:05:13,045::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:05:13,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159762 s, 256 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:05:13,396::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:05:13,402::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:05:13,403::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:05:21,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:05:21,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074624 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:05:23,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:05:23,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844555 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:05:31,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:05:31,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860016 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:05:33,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:05:33,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880083 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:05:41,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:05:41,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000939654 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:05:43,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:05:43,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132859 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:05:51,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:05:51,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808831 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:05:53,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:05:53,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000695936 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:06:01,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:06:01,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010979 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:06:03,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:06:03,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148153 s, 276 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:06:11,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:06:11,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00731177 s, 560 kB/s\n'; <rc> = 0
>Thread-13845::WARNING::2013-03-17 06:06:13,433::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:06:13,439::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 06:06:13,519::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 06:06:13,521::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 06:06:13,522::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:06:13,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0014433 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:06:13,561::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0014433 s, 1.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 06:06:13,562::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 06:06:13,562::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 06:06:13,564::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 06:06:13,565::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:06:13,565::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:06:13,566::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:06:13,566::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:06:13,567::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:06:13,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:06:14,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:06:14,060::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:06:14,061::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 06:06:14,063::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 06:06:14,064::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 06:06:14,065::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 06:06:14,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:06:14,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:06:14,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:06:14,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010103 s, 405 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:06:21,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:06:21,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852541 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:06:23,442::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:06:23,443::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:06:23,864::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:06:23,870::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:06:23,872::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:06:24,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:06:24,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796609 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:06:31,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:06:31,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000974154 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:06:34,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:06:34,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110352 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:06:41,898::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 06:06:41,900::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 06:06:41,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:06:41,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00244844 s, 836 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:06:41,948::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00244844 s, 836 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 06:06:41,948::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 06:06:41,948::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 06:06:41,950::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 06:06:41,951::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:06:41,951::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:06:41,952::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:06:41,952::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:06:41,952::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:06:41,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:06:42,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:06:42,355::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:06:42,356::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 06:06:42,359::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 06:06:42,360::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 06:06:42,360::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 06:06:42,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:06:42,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:06:42,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:06:42,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013761 s, 298 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:06:44,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:06:44,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000736679 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:06:52,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:06:52,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134563 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:06:54,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:06:54,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137568 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:07:02,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:07:02,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00203058 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:07:04,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:07:04,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151875 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:07:12,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:07:12,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112839 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:07:14,944::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:07:14,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147602 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:07:22,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:07:23,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079253 s, 5.2 MB/s\n'; <rc> = 0
>Thread-13846::WARNING::2013-03-17 06:07:23,897::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:07:23,901::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 06:07:25,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:07:25,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843979 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:07:33,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:07:33,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0045451 s, 901 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:07:33,903::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 06:07:33,904::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:07:33,906::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:07:34,332::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:07:34,339::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:07:34,341::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:07:35,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:07:35,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834151 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:07:43,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:07:43,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851195 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:07:45,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:07:45,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784855 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:07:53,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:07:53,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131043 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:07:55,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:07:55,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158277 s, 259 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:08:03,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:08:03,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000991789 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:08:05,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:08:05,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147785 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:08:13,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:08:13,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117111 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:08:15,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:08:15,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743424 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:08:23,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:08:23,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906726 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:08:25,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:08:25,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000970606 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:08:33,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:08:33,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114622 s, 3.6 MB/s\n'; <rc> = 0
>Thread-13847::WARNING::2013-03-17 06:08:34,369::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:08:34,375::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 06:08:35,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:08:35,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870003 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:08:43,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:08:43,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014439 s, 2.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:08:44,379::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:08:44,381::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:08:44,830::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:08:44,837::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:08:44,838::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:08:45,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:08:45,744::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00939853 s, 436 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:08:53,741::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:08:53,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149427 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:08:55,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:08:55,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861223 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:09:03,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:09:03,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842499 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:09:05,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:09:05,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000934078 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:09:13,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:09:13,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169765 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:09:15,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:09:15,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820331 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:09:23,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:09:23,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174449 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:09:26,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:09:26,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000945291 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:09:33,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:09:33,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101978 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:09:36,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:09:36,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879162 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:09:44,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:09:44,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00936834 s, 437 kB/s\n'; <rc> = 0
>Thread-13848::WARNING::2013-03-17 06:09:44,861::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:09:44,867::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 06:09:46,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:09:46,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0048833 s, 839 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:09:54,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:09:54,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831761 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:09:54,874::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:09:54,875::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:09:55,341::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:09:55,348::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:09:55,350::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:09:56,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:09:56,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160726 s, 255 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:10:04,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:10:04,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106712 s, 384 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:10:06,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:10:06,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175544 s, 233 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:10:14,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:10:14,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000952446 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:10:16,414::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:10:16,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830111 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:10:24,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:10:24,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119811 s, 342 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:10:26,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:10:26,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150063 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:10:34,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:10:34,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150381 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:10:36,556::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:10:36,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852212 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:10:44,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:10:44,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123721 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:10:46,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:10:46,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790743 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:10:54,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:10:54,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757685 s, 5.4 MB/s\n'; <rc> = 0
>Thread-13849::WARNING::2013-03-17 06:10:55,376::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:10:55,380::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 06:10:56,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:10:56,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890737 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:11:04,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:11:04,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766226 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:11:05,382::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:11:05,384::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:11:05,804::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:11:05,810::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:11:05,811::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:11:06,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:11:06,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152324 s, 269 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:11:14,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:11:14,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847011 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:11:16,888::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 06:11:16,889::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 06:11:16,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:11:16,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00089285 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:11:16,919::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00089285 s, 2.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 06:11:16,920::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 06:11:16,920::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 06:11:16,922::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 06:11:16,923::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:11:16,924::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:11:16,924::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:11:16,925::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:11:16,925::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:11:16,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:11:17,365::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:11:17,369::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:11:17,370::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 06:11:17,373::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 06:11:17,375::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 06:11:17,376::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 06:11:17,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:11:17,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:11:17,760::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:11:17,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0157659 s, 260 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:11:24,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:11:24,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809441 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:11:27,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:11:27,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822615 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:11:34,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:11:34,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146746 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:11:37,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:11:37,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103021 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:11:44,924::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 06:11:44,925::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 06:11:44,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:11:44,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00304695 s, 672 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:11:44,973::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00304695 s, 672 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 06:11:44,974::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 06:11:44,974::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 06:11:44,975::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 06:11:44,976::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:11:44,977::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:11:44,977::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:11:44,978::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:11:44,978::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:11:44,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:11:45,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:11:45,401::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:11:45,402::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 06:11:45,404::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 06:11:45,405::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 06:11:45,406::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 06:11:45,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:11:45,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:11:45,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:11:45,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111938 s, 366 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:11:47,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:11:48,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000732178 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:11:55,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:11:55,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137436 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:11:58,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:11:58,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147232 s, 2.8 MB/s\n'; <rc> = 0
>Thread-13850::WARNING::2013-03-17 06:12:05,840::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:12:05,843::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:12:05,863::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:12:05,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755745 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:12:08,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:12:08,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868292 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:12:15,845::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:12:15,846::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:12:15,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:12:15,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133961 s, 306 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:12:16,291::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:12:16,297::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:12:16,298::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:12:18,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:12:18,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798241 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:12:25,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:12:25,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913752 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:12:28,281::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:12:28,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00413129 s, 991 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:12:36,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:12:36,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147491 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:12:38,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:12:38,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000976587 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:12:46,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:12:46,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159901 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:12:48,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:12:48,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838589 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:12:56,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:12:56,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165829 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:12:58,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:12:58,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136746 s, 300 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:13:06,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:13:06,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106763 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:13:08,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:13:08,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017238 s, 238 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:13:16,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:13:16,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00499444 s, 820 kB/s\n'; <rc> = 0
>Thread-13851::WARNING::2013-03-17 06:13:16,345::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:13:16,349::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 06:13:18,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:13:18,756::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917288 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:13:26,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:13:26,299::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132147 s, 310 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:13:26,351::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 06:13:26,353::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:13:26,354::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:13:26,824::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:13:26,831::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:13:26,832::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:13:28,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:13:28,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163227 s, 251 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:13:36,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:13:36,333::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763834 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:13:38,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:13:38,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000706576 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:13:46,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:13:46,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846549 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:13:48,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:13:48,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000969321 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:13:56,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:13:56,409::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790358 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:13:59,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:13:59,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144731 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:14:06,414::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:14:06,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00295743 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:14:09,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:14:09,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138388 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:14:16,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:14:16,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833395 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:14:19,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:14:19,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000718026 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:14:26,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:14:26,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000676656 s, 6.1 MB/s\n'; <rc> = 0
>Thread-13852::WARNING::2013-03-17 06:14:26,859::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:14:26,863::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 06:14:29,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:14:29,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145559 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:14:36,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:14:36,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147288 s, 278 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:14:36,866::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:14:36,868::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:14:37,318::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:14:37,324::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:14:37,326::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:14:39,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:14:39,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115051 s, 356 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:14:46,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:14:46,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173219 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:14:49,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:14:49,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155401 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:14:56,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:14:56,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135249 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:14:59,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:14:59,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000709109 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:15:06,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:15:06,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108344 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:15:09,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:15:09,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171532 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:15:16,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:15:16,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136657 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:15:19,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:15:19,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859933 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:15:26,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:15:26,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139894 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:15:29,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:15:29,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114687 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:15:36,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:15:36,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760712 s, 5.4 MB/s\n'; <rc> = 0
>Thread-13853::WARNING::2013-03-17 06:15:37,361::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:15:37,365::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 06:15:39,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:15:39,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741607 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:15:46,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:15:47,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0394895 s, 104 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:15:47,368::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:15:47,369::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:15:47,804::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:15:47,810::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:15:47,812::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:15:49,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:15:49,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197628 s, 207 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:15:57,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:15:57,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103335 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:15:59,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:16:00,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000996055 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:16:07,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:16:07,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800092 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:16:10,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:16:10,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123592 s, 331 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:16:17,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:16:17,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132983 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:16:20,127::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 06:16:20,129::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 06:16:20,129::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:16:20,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00138738 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:16:20,170::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00138738 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 06:16:20,171::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 06:16:20,171::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 06:16:20,173::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 06:16:20,174::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:16:20,174::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:16:20,175::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:16:20,175::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:16:20,176::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:16:20,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:16:20,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:16:20,585::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:16:20,586::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 06:16:20,591::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 06:16:20,593::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 06:16:20,593::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 06:16:20,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:16:20,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:16:20,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:16:20,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00699385 s, 586 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:16:27,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:16:27,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193743 s, 211 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:16:31,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:16:31,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137992 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:16:37,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:16:37,309::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013217 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:16:41,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:16:41,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106978 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:16:47,314::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 06:16:47,318::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 06:16:47,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:16:47,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00144985 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:16:47,379::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00144985 s, 1.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 06:16:47,381::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 06:16:47,383::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 06:16:47,388::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 06:16:47,394::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:16:47,398::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:16:47,400::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:16:47,402::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:16:47,404::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:16:47,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-13854::WARNING::2013-03-17 06:16:47,849::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:16:47,855::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:16:47,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:16:47,875::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:16:47,876::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 06:16:47,878::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 06:16:47,879::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 06:16:47,880::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 06:16:47,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:16:48,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:16:48,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:16:48,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163392 s, 251 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:16:51,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:16:51,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867439 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:16:57,858::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:16:57,859::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:16:58,264::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:16:58,270::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:16:58,272::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:16:58,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:16:58,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117633 s, 348 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:17:01,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:17:01,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177855 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:17:08,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:17:08,382::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014618 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:17:11,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:17:11,365::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811538 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:17:18,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:17:18,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891619 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:17:21,405::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:17:21,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829608 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:17:28,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:17:28,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149493 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:17:31,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:17:31,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081651 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:17:38,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:17:38,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790061 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:17:41,556::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:17:41,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154974 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:17:48,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:17:48,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803654 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:17:51,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:17:51,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0074334 s, 551 kB/s\n'; <rc> = 0
>Thread-13855::WARNING::2013-03-17 06:17:58,325::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:17:58,329::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:17:58,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:17:58,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872463 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:18:01,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:18:01,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00200485 s, 2.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:18:08,331::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:18:08,333::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:18:08,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:18:08,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146781 s, 279 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:18:08,834::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:18:08,839::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:18:08,841::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:18:11,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:18:11,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161328 s, 254 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:18:18,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:18:18,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154309 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:18:22,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:18:22,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880072 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:18:28,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:18:28,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842553 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:18:32,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:18:32,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136657 s, 300 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:18:38,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:18:38,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160755 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:18:42,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:18:42,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912135 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:18:48,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:18:48,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001654 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:18:52,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:18:52,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00770621 s, 532 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:18:58,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:18:58,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001432 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:19:02,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:19:02,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135316 s, 3.0 MB/s\n'; <rc> = 0
>Thread-13856::WARNING::2013-03-17 06:19:08,885::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:19:08,889::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:19:08,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:19:08,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165493 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:19:12,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:19:12,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827351 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:19:18,891::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 06:19:18,892::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:19:18,893::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:19:18,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:19:19,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184004 s, 223 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:19:19,342::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:19:19,348::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:19:19,349::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:19:22,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:19:22,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00887528 s, 462 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:19:29,021::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:19:29,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184107 s, 222 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:19:32,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:19:32,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124477 s, 329 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:19:39,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:19:39,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142851 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:19:42,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:19:42,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888091 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:19:49,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:19:49,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810538 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:19:52,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:19:52,801::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0094106 s, 435 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:19:59,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:19:59,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936924 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:20:02,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:20:02,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144977 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:20:09,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:20:09,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146265 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:20:12,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:20:12,963::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117035 s, 350 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:20:19,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:20:19,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753043 s, 5.4 MB/s\n'; <rc> = 0
>Thread-13857::WARNING::2013-03-17 06:20:19,371::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:20:19,375::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 06:20:23,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:20:23,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759427 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:20:29,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:20:29,392::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:20:29,406::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:20:29,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142453 s, 288 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:20:29,887::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:20:29,892::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:20:29,893::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:20:33,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:20:33,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120636 s, 340 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:20:39,445::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:20:39,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157475 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:20:43,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:20:43,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916597 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:20:49,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:20:49,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768215 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:20:53,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:20:53,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146241 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:20:59,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:20:59,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000717558 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:21:03,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:21:03,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883887 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:21:09,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:21:09,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000945426 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:21:13,372::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:21:13,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017852 s, 229 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:21:19,605::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:21:19,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878636 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:21:23,462::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 06:21:23,463::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 06:21:23,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:21:23,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00121973 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:21:23,501::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00121973 s, 1.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 06:21:23,501::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 06:21:23,501::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 06:21:23,503::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 06:21:23,507::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:21:23,508::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:21:23,508::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:21:23,509::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:21:23,509::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:21:23,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:21:23,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:21:23,956::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:21:23,957::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 06:21:23,959::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 06:21:23,961::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 06:21:23,962::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 06:21:23,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:21:24,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:21:24,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:21:24,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00748639 s, 547 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:21:29,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:21:29,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00354652 s, 1.2 MB/s\n'; <rc> = 0
>Thread-13858::WARNING::2013-03-17 06:21:29,917::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:21:29,921::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 06:21:34,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:21:34,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000961577 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:21:39,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:21:39,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186234 s, 220 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:21:39,924::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:21:39,925::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:21:40,349::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:21:40,356::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:21:40,357::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:21:44,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:21:44,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120485 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:21:49,783::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 06:21:49,784::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 06:21:49,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:21:49,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000803201 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:21:49,816::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000803201 s, 2.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 06:21:49,816::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 06:21:49,816::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 06:21:49,818::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 06:21:49,819::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:21:49,820::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:21:49,820::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:21:49,820::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:21:49,821::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:21:49,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:21:50,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:21:50,210::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:21:50,211::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 06:21:50,214::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 06:21:50,215::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 06:21:50,216::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 06:21:50,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:21:50,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:21:50,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:21:50,631::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203517 s, 201 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:21:54,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:21:54,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758947 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:22:00,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:22:00,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075186 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:22:04,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:22:04,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824241 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:22:10,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:22:10,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793803 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:22:14,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:22:14,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829386 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:22:20,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:22:20,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011755 s, 348 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:22:24,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:22:24,809::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178795 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:22:30,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:22:30,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916571 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:22:34,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:22:34,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149491 s, 2.7 MB/s\n'; <rc> = 0
>Thread-13859::WARNING::2013-03-17 06:22:40,397::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:22:40,400::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:22:40,796::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:22:40,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000905317 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:22:44,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:22:44,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158488 s, 2.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:22:50,402::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:22:50,404::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:22:50,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:22:50,878::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-22::DEBUG::2013-03-17 06:22:50,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141598 s, 289 kB/s\n'; <rc> = 0
>Thread-21::WARNING::2013-03-17 06:22:50,923::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:22:50,924::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:22:55,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:22:55,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741047 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:23:00,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:23:00,950::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815538 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:23:05,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:23:05,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0245021 s, 167 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:23:10,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:23:10,992::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010605 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:23:15,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:23:15,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00257635 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:23:20,997::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:23:21,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134604 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:23:25,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:23:25,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817244 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:23:31,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:23:31,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161586 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:23:35,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:23:35,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154677 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:23:41,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:23:41,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169316 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:23:45,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:23:45,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000914079 s, 4.5 MB/s\n'; <rc> = 0
>Thread-13860::WARNING::2013-03-17 06:23:50,951::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:23:50,954::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:23:51,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:23:51,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143199 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:23:55,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:23:55,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172669 s, 2.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:24:00,959::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:24:00,960::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:24:01,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:24:01,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01473 s, 278 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:24:01,452::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:24:01,458::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:24:01,459::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:24:05,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:24:05,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00956393 s, 428 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:24:11,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:24:11,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104105 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:24:15,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:24:15,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156299 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:24:21,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:24:21,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791338 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:24:25,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:24:25,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809666 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:24:31,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:24:31,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875109 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:24:35,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:24:35,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000711051 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:24:41,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:24:41,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00705018 s, 581 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:24:45,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:24:45,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853557 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:24:51,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:24:51,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127841 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:24:56,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:24:56,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181603 s, 2.3 MB/s\n'; <rc> = 0
>Thread-13861::WARNING::2013-03-17 06:25:01,485::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:25:01,488::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:25:01,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:25:01,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860592 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:25:06,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:25:06,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782577 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:25:11,492::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 06:25:11,493::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:25:11,495::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:25:11,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:25:11,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0202573 s, 202 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:25:11,938::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:25:11,943::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:25:11,945::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:25:16,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:25:16,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101747 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:25:21,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:25:21,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00070518 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:25:26,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:25:26,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757485 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:25:31,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:25:31,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789877 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:25:36,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:25:36,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869037 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:25:41,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:25:41,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00654051 s, 626 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:25:46,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:25:46,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771746 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:25:51,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:25:51,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125376 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:25:56,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:25:56,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143914 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:26:01,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:26:01,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137051 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:26:06,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:26:06,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00372852 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:26:11,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:26:11,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00355964 s, 1.2 MB/s\n'; <rc> = 0
>Thread-13862::WARNING::2013-03-17 06:26:11,973::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:26:11,980::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 06:26:16,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:26:16,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104375 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:26:21,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:26:21,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139309 s, 294 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:26:21,983::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:26:21,985::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:26:22,454::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:26:22,461::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:26:22,462::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:26:26,891::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 06:26:26,894::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 06:26:26,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:26:26,958::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00129133 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:26:26,959::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00129133 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 06:26:26,960::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 06:26:26,960::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 06:26:26,962::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 06:26:26,963::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:26:26,963::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:26:26,964::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:26:26,964::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:26:26,965::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:26:26,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:26:27,401::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:26:27,406::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:26:27,407::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 06:26:27,409::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 06:26:27,410::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 06:26:27,411::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 06:26:27,412::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:26:27,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:26:27,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:26:27,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0257762 s, 159 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:26:31,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:26:32,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0019516 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:26:37,863::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:26:37,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000708152 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:26:42,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:26:42,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135813 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:26:47,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:26:47,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143939 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:26:52,063::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 06:26:52,064::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 06:26:52,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:26:52,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000874831 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:26:52,096::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000874831 s, 2.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 06:26:52,096::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 06:26:52,097::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 06:26:52,099::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 06:26:52,100::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:26:52,101::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:26:52,102::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:26:52,102::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:26:52,103::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:26:52,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:26:52,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:26:52,544::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:26:52,545::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 06:26:52,548::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 06:26:52,549::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 06:26:52,549::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 06:26:52,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:26:52,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:26:52,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:26:52,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0200199 s, 205 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:26:58,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:26:58,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00686121 s, 597 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:27:02,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:27:03,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139172 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:27:08,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:27:08,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014956 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:27:13,019::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:27:13,046::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800006 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:27:18,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:27:18,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132605 s, 3.1 MB/s\n'; <rc> = 0
>Thread-13863::WARNING::2013-03-17 06:27:22,492::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:27:22,496::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:27:23,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:27:23,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835706 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:27:28,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:27:28,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142691 s, 2.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:27:32,498::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:27:32,499::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:27:32,921::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:27:32,928::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:27:32,929::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:27:33,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:27:33,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168486 s, 243 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:27:38,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:27:38,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0233555 s, 175 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:27:43,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:27:43,188::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166328 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:27:48,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:27:48,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171467 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:27:53,194::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:27:53,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812613 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:27:58,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:27:58,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873379 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:28:03,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:28:03,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882027 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:28:08,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:28:08,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081528 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:28:13,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:28:13,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136293 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:28:18,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:28:18,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915997 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:28:23,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:28:23,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012747 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:28:28,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:28:28,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181911 s, 2.3 MB/s\n'; <rc> = 0
>Thread-13864::WARNING::2013-03-17 06:28:32,958::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:28:32,960::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:28:33,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:28:33,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797091 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:28:38,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:28:38,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746348 s, 5.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:28:42,962::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:28:42,964::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:28:43,385::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:28:43,392::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:28:43,394::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:28:43,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:28:43,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136953 s, 299 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:28:48,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:28:48,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00234501 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:28:53,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:28:53,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00067772 s, 6.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:28:58,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:28:59,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832751 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:29:03,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:29:03,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838653 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:29:09,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:29:09,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141785 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:29:13,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:29:13,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00208487 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:29:19,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:29:19,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076863 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:29:23,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:29:23,631::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144112 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:29:29,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:29:29,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00419947 s, 975 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:29:33,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:29:33,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014384 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:29:39,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:29:39,347::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135122 s, 3.0 MB/s\n'; <rc> = 0
>Thread-13865::WARNING::2013-03-17 06:29:43,459::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:29:43,464::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:29:43,685::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:29:43,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185216 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:29:49,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:29:49,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793126 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:29:53,466::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:29:53,468::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:29:53,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:29:53,786::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130427 s, 314 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:29:53,944::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:29:53,950::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:29:53,951::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:29:59,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:29:59,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862906 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:30:03,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:30:03,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874157 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:30:09,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:30:09,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101546 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:30:13,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:30:13,866::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140363 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:30:19,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:30:19,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122522 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:30:23,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:30:23,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109083 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:30:29,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:30:29,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154584 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:30:33,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:30:33,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140352 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:30:39,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:30:39,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790624 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:30:43,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:30:43,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913499 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:30:49,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:30:49,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136275 s, 3.0 MB/s\n'; <rc> = 0
>Thread-13866::WARNING::2013-03-17 06:30:53,978::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:30:53,981::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:30:53,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:30:54,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787067 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:31:00,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:31:00,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852372 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:31:03,983::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 06:31:03,984::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:31:03,985::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:31:04,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:31:04,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102917 s, 398 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:31:04,403::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:31:04,409::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:31:04,411::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:31:10,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:31:10,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143754 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:31:14,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:31:14,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843272 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:31:20,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:31:20,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156272 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:31:24,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:31:24,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00631577 s, 649 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:31:30,319::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 06:31:30,320::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 06:31:30,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:31:30,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00080406 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:31:30,363::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00080406 s, 2.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 06:31:30,363::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 06:31:30,364::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 06:31:30,365::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 06:31:30,366::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:31:30,367::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:31:30,368::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:31:30,368::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:31:30,368::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:31:30,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:31:30,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:31:30,816::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:31:30,817::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 06:31:30,820::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 06:31:30,821::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 06:31:30,821::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 06:31:30,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:31:31,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:31:31,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:31:31,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183883 s, 223 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:31:34,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:31:34,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0044356 s, 923 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:31:41,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:31:41,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083368 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:31:44,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:31:44,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138824 s, 295 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:31:51,482::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:31:51,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0250242 s, 164 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:31:54,264::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 06:31:54,269::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 06:31:54,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:31:54,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00115134 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:31:54,305::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00115134 s, 1.8 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 06:31:54,305::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 06:31:54,305::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 06:31:54,307::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 06:31:54,308::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:31:54,309::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:31:54,309::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:31:54,310::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:31:54,310::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:31:54,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:31:54,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:31:54,752::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:31:54,752::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 06:31:54,755::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 06:31:54,756::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 06:31:54,756::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 06:31:54,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:31:55,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:31:55,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:31:55,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142063 s, 288 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:32:01,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:32:01,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139257 s, 2.9 MB/s\n'; <rc> = 0
>Thread-13867::WARNING::2013-03-17 06:32:04,439::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:32:04,444::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:32:05,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:32:05,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885057 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:32:11,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:32:11,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828684 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:32:14,447::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:32:14,449::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:32:14,881::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:32:14,888::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:32:14,889::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:32:15,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:32:15,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151394 s, 271 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:32:21,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:32:21,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000707395 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:32:25,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:32:25,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769451 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:32:31,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:32:31,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813723 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:32:35,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:32:35,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167131 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:32:41,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:32:41,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771973 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:32:45,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:32:45,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155765 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:32:51,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:32:52,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148059 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:32:55,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:32:55,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728162 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:33:02,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:33:02,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169658 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:33:05,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:33:05,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144513 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:33:12,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:33:12,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153038 s, 2.7 MB/s\n'; <rc> = 0
>Thread-13868::WARNING::2013-03-17 06:33:14,917::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:33:14,919::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:33:15,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:33:15,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00906692 s, 452 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:33:22,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:33:22,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000962492 s, 4.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:33:24,922::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:33:24,923::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:33:25,334::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:33:25,340::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:33:25,342::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:33:25,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:33:25,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116086 s, 353 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:33:32,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:33:32,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000933093 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:33:35,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:33:35,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00621691 s, 659 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:33:42,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:33:42,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00200627 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:33:45,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:33:45,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158416 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:33:52,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:33:52,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015581 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:33:55,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:33:55,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113018 s, 362 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:34:02,522::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:34:02,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817191 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:34:05,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:34:05,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136862 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:34:12,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:34:12,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149398 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:34:15,740::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:34:15,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135953 s, 301 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:34:22,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:34:22,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110532 s, 3.7 MB/s\n'; <rc> = 0
>Thread-13869::WARNING::2013-03-17 06:34:25,389::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:34:25,394::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:34:25,797::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:34:25,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000732116 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:34:32,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:34:32,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831133 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:34:35,405::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:34:35,406::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:34:35,814::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:34:35,820::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:34:35,822::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:34:35,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:34:35,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00309715 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:34:42,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:34:42,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142915 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:34:45,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:34:45,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851672 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:34:52,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:34:52,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785774 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:34:55,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:34:55,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800059 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:35:02,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:35:02,992::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0037988 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:35:05,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:35:06,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146098 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:35:13,033::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:35:13,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836948 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:35:16,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:35:16,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00322906 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:35:23,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:35:23,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108843 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:35:26,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:35:26,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151832 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:35:33,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:35:33,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0028731 s, 1.4 MB/s\n'; <rc> = 0
>Thread-13870::WARNING::2013-03-17 06:35:35,880::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:35:35,883::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:35:36,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:35:36,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083172 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:35:43,260::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:35:43,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784978 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:35:45,897::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:35:45,898::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:35:46,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:35:46,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140728 s, 291 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:35:46,375::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:35:46,381::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:35:46,382::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:35:53,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:35:53,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753709 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:35:56,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:35:56,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000926065 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:36:03,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:36:03,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156465 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:36:06,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:36:06,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846089 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:36:13,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:36:13,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000940907 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:36:16,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:36:16,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916522 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:36:23,605::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:36:23,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000666552 s, 6.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:36:26,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:36:26,365::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156068 s, 262 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:36:33,691::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 06:36:33,692::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 06:36:33,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:36:33,734::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00150073 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:36:33,735::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00150073 s, 1.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 06:36:33,736::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 06:36:33,736::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 06:36:33,738::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 06:36:33,743::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:36:33,744::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:36:33,744::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:36:33,745::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:36:33,745::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:36:33,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:36:34,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:36:34,182::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:36:34,183::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 06:36:34,185::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 06:36:34,187::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 06:36:34,187::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 06:36:34,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:36:34,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:36:34,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:36:34,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0068597 s, 597 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:36:36,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:36:36,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00707071 s, 579 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:36:44,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:36:44,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132901 s, 308 kB/s\n'; <rc> = 0
>Thread-13871::WARNING::2013-03-17 06:36:46,410::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:36:46,413::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:36:46,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:36:46,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000922585 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:36:55,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:36:55,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870146 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:36:56,415::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 06:36:56,416::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:36:56,417::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:36:56,455::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 06:36:56,458::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 06:36:56,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:36:56,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0269061 s, 76.1 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:36:56,520::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0269061 s, 76.1 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 06:36:56,520::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 06:36:56,521::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 06:36:56,521::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 06:36:56,523::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm reload operation' is holding the operation mutex, waiting...
>Thread-21::DEBUG::2013-03-17 06:36:56,989::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:36:56,995::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:36:56,996::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:36:57,003::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:36:57,006::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:36:57,006::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:36:57,007::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:36:57,007::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:36:57,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:36:57,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:36:57,392::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:36:57,393::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 06:36:57,395::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 06:36:57,396::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 06:36:57,397::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 06:36:57,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:36:57,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:36:57,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:36:57,801::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187433 s, 219 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:37:05,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:37:05,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868963 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:37:07,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:37:07,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0285496 s, 143 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:37:15,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:37:15,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000918405 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:37:17,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:37:17,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169207 s, 242 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:37:25,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:37:25,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787563 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:37:27,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:37:27,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151688 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:37:35,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:37:35,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0209529 s, 195 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:37:37,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:37:38,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0326998 s, 125 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:37:45,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:37:45,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142588 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:37:48,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:37:48,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895392 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:37:55,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:37:55,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113123 s, 362 kB/s\n'; <rc> = 0
>Thread-13872::WARNING::2013-03-17 06:37:57,059::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:37:57,064::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:37:58,118::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:37:58,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148542 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:38:05,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:38:05,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157383 s, 2.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:38:07,068::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:38:07,070::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:38:07,484::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:38:07,491::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:38:07,492::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:38:08,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:38:08,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178083 s, 230 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:38:15,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:38:16,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000952474 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:38:18,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:38:18,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011014 s, 372 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:38:26,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:38:26,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865317 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:38:28,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:38:28,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084128 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:38:36,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:38:36,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770715 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:38:38,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:38:38,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154755 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:38:46,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:38:46,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206976 s, 198 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:38:48,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:38:48,382::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00094547 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:38:56,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:38:56,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140908 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:38:58,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:38:58,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163716 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:39:06,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:39:06,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001744 s, 2.3 MB/s\n'; <rc> = 0
>Thread-13873::WARNING::2013-03-17 06:39:07,520::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:39:07,526::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:39:08,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:39:08,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000934577 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:39:16,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:39:16,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000964907 s, 4.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:39:17,530::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:39:17,532::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:39:17,958::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:39:17,965::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:39:17,966::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:39:18,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:39:18,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177929 s, 230 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:39:26,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:39:26,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0018564 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:39:28,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:39:28,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764457 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:39:36,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:39:36,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00290657 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:39:38,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:39:38,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000977625 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:39:46,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:39:46,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143107 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:39:48,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:39:48,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808791 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:39:56,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:39:56,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810041 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:39:58,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:39:58,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130019 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:40:06,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:40:06,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791866 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:40:08,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:40:08,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148009 s, 277 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:40:16,950::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:40:16,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013502 s, 3.0 MB/s\n'; <rc> = 0
>Thread-13874::WARNING::2013-03-17 06:40:17,994::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:40:18,005::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:40:18,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:40:18,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795658 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:40:27,118::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:40:27,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175813 s, 2.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:40:28,008::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:40:28,010::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:40:28,460::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:40:28,467::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:40:28,469::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:40:28,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:40:28,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161944 s, 253 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:40:37,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:40:37,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812692 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:40:38,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:40:38,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154035 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:40:47,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:40:47,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932576 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:40:48,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:40:48,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130375 s, 314 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:40:57,350::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:40:57,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814219 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:40:58,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:40:58,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119454 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:41:07,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:41:07,448::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790463 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:41:08,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:41:08,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120112 s, 341 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:41:17,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:41:17,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157456 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:41:19,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:41:19,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886733 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:41:27,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:41:27,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160787 s, 2.5 MB/s\n'; <rc> = 0
>Thread-13875::WARNING::2013-03-17 06:41:28,512::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:41:28,514::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:41:29,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:41:29,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143535 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:41:37,860::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 06:41:37,868::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 06:41:37,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:41:37,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0113599 s, 180 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:41:37,921::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0113599 s, 180 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 06:41:37,921::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 06:41:37,922::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 06:41:37,924::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 06:41:37,925::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:41:37,925::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:41:37,926::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:41:37,926::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:41:37,927::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:41:37,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:41:38,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:41:38,320::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:41:38,321::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 06:41:38,324::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 06:41:38,325::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 06:41:38,326::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 06:41:38,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:41:38,518::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:41:38,521::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:41:38,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:41:38,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:41:38,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109772 s, 373 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:41:39,051::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:41:39,057::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:41:39,058::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:41:39,099::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:41:39,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145076 s, 282 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:41:48,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:41:48,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886736 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:41:49,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:41:49,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00441631 s, 927 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:41:58,859::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:41:58,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924843 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:41:59,205::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 06:41:59,207::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 06:41:59,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:41:59,236::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000962105 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:41:59,238::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000962105 s, 2.1 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 06:41:59,239::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 06:41:59,239::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 06:41:59,244::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 06:41:59,245::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:41:59,245::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:41:59,246::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:41:59,246::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:41:59,247::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:41:59,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:41:59,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:41:59,650::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:41:59,650::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 06:41:59,653::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 06:41:59,654::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 06:41:59,654::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 06:41:59,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:42:00,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:42:00,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:42:00,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911339 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:42:08,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:42:08,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000739026 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:42:10,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:42:10,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0083951 s, 488 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:42:19,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:42:19,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764565 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:42:20,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:42:20,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151655 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:42:29,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:42:29,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166006 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:42:30,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:42:30,195::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00567193 s, 722 kB/s\n'; <rc> = 0
>Thread-13876::WARNING::2013-03-17 06:42:39,084::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:42:39,091::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 06:42:39,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:42:39,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799128 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:42:40,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:42:40,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814656 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:42:49,093::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 06:42:49,095::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:42:49,096::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:42:49,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:42:49,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108062 s, 379 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:42:49,575::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:42:49,581::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:42:49,582::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:42:50,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:42:50,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177637 s, 231 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:42:59,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:42:59,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829647 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:43:00,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:43:00,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0026355 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:43:09,415::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:43:09,448::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893894 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:43:10,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:43:10,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107988 s, 379 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:43:19,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:43:19,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082582 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:43:20,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:43:20,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113138 s, 362 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:43:29,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:43:29,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820217 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:43:30,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:43:30,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087497 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:43:39,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:43:39,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080464 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:43:40,465::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:43:40,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0589295 s, 69.5 kB/s\n'; <rc> = 0
>Thread-13877::WARNING::2013-03-17 06:43:49,612::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:43:49,617::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 06:43:49,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:43:49,756::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017936 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:43:50,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:43:50,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823559 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:43:59,619::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:43:59,621::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:43:59,797::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:43:59,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167435 s, 2.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:44:00,066::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:44:00,071::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:44:00,073::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:44:00,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:44:00,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124681 s, 329 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:44:09,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:44:09,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008272 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:44:10,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:44:10,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146062 s, 280 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:44:19,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:44:20,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154414 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:44:20,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:44:20,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134182 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:44:30,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:44:30,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137706 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:44:30,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:44:30,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890087 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:44:40,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:44:40,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154905 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:44:40,792::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:44:40,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832435 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:44:50,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:44:50,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087821 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:44:50,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:44:50,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136141 s, 3.0 MB/s\n'; <rc> = 0
>Thread-13878::WARNING::2013-03-17 06:45:00,099::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:45:00,102::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 06:45:00,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:45:00,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00284633 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:45:00,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:45:00,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869458 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:45:10,107::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:45:10,109::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:45:10,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:45:10,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110415 s, 3.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:45:10,571::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:45:10,577::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:45:10,578::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:45:10,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:45:10,958::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173236 s, 236 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:45:20,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:45:20,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008642 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:45:20,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:45:20,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074283 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:45:30,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:45:30,561::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133894 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:45:30,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:45:31,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137371 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:45:40,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:45:40,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804097 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:45:41,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:45:41,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000960659 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:45:50,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:45:50,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127357 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:45:51,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:45:51,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159175 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:46:00,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:46:00,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808713 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:46:01,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:46:01,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148287 s, 2.8 MB/s\n'; <rc> = 0
>Thread-13879::WARNING::2013-03-17 06:46:10,606::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:46:10,611::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 06:46:10,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:46:10,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161036 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:46:11,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:46:11,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149094 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:46:20,620::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:46:20,621::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:46:20,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:46:20,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764548 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:46:21,075::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:46:21,080::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:46:21,081::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:46:21,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:46:21,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.012516 s, 327 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:46:30,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:46:31,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014599 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:46:31,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:46:31,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171956 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:46:41,061::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 06:46:41,063::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 06:46:41,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:46:41,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00141021 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:46:41,104::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00141021 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 06:46:41,105::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 06:46:41,105::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 06:46:41,107::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 06:46:41,108::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:46:41,108::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:46:41,109::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:46:41,109::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:46:41,109::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:46:41,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:46:41,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:46:41,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00369989 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:46:41,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:46:41,580::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:46:41,581::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 06:46:41,583::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 06:46:41,584::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 06:46:41,584::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 06:46:41,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:46:42,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:46:42,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:46:42,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143541 s, 285 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:46:51,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:46:51,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000958041 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:46:52,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:46:52,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865633 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:47:01,466::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 06:47:01,467::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 06:47:01,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:47:01,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00145062 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:47:01,518::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00145062 s, 1.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 06:47:01,518::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 06:47:01,518::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 06:47:01,520::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 06:47:01,521::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:47:01,521::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:47:01,522::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:47:01,522::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:47:01,522::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:47:01,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:47:01,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:47:01,939::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:47:01,940::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 06:47:01,942::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 06:47:01,943::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 06:47:01,944::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 06:47:01,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:47:02,178::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:47:02,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139992 s, 293 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:47:02,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:47:02,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:47:02,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142104 s, 288 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:47:12,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:47:12,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080074 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:47:12,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:47:12,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850087 s, 4.8 MB/s\n'; <rc> = 0
>Thread-13880::WARNING::2013-03-17 06:47:21,107::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:47:21,111::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 06:47:22,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:47:22,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884461 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:47:22,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:47:22,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907616 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:47:31,113::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:47:31,115::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:47:31,594::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:47:31,601::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:47:31,602::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:47:32,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:47:32,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131116 s, 312 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:47:32,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:47:32,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173681 s, 236 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:47:42,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:47:42,647::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:47:42,700::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894946 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:47:42,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113139 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:47:52,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:47:52,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892939 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:47:52,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:47:52,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000939938 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:48:02,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:48:02,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847989 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:48:02,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:48:02,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853169 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:48:12,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:48:12,819::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151976 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:48:12,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:48:12,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00315455 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:48:22,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:48:22,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017744 s, 231 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:48:22,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:48:23,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143578 s, 2.9 MB/s\n'; <rc> = 0
>Thread-13881::WARNING::2013-03-17 06:48:31,644::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:48:31,652::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:48:32,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:48:32,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102027 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:48:33,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:48:33,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855166 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:48:41,654::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 06:48:41,656::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:48:41,657::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:48:42,091::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:48:42,098::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:48:42,100::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:48:42,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:48:42,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168951 s, 242 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:48:43,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:48:43,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144583 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:48:52,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:48:53,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909721 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:48:53,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:48:53,236::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811897 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:49:03,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:49:03,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153491 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:49:03,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:49:03,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809001 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:49:13,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:49:13,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809014 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:49:13,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:49:13,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936498 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:49:23,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:49:23,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848369 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:49:23,414::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:49:23,439::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820569 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:49:33,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:49:33,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842436 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:49:33,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:49:33,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015011 s, 2.7 MB/s\n'; <rc> = 0
>Thread-13882::WARNING::2013-03-17 06:49:42,126::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:49:42,130::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:49:43,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:49:43,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779838 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:49:43,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:49:43,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853437 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:49:52,132::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:49:52,133::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:49:52,585::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:49:52,591::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:49:52,593::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:49:53,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:49:53,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158831 s, 258 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:49:53,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:49:53,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173941 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:50:03,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:50:03,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161427 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:50:03,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:50:03,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150924 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:50:13,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:50:13,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139926 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:50:13,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:50:13,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737886 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:50:23,352::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:50:23,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855969 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:50:23,875::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:50:23,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152488 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:50:33,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:50:33,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141557 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:50:33,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:50:34,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153045 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:50:43,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:50:43,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165413 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:50:44,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:50:44,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015305 s, 2.7 MB/s\n'; <rc> = 0
>Thread-13883::WARNING::2013-03-17 06:50:52,636::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:50:52,640::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:50:53,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:50:53,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831438 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:50:54,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:50:54,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175388 s, 2.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:51:02,642::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:51:02,643::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:51:03,117::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:51:03,124::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:51:03,125::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:51:03,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:51:03,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177522 s, 231 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:51:04,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:51:04,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015398 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:51:13,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:51:13,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800802 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:51:14,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:51:14,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874385 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:51:23,603::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:51:23,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148921 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:51:24,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:51:24,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791979 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:51:33,648::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:51:33,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00510222 s, 803 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:51:34,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:51:34,484::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125668 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:51:43,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:51:43,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152835 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:51:44,522::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 06:51:44,524::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 06:51:44,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:51:44,561::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00906912 s, 226 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:51:44,563::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00906912 s, 226 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 06:51:44,564::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 06:51:44,564::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 06:51:44,565::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 06:51:44,566::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:51:44,567::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:51:44,568::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:51:44,568::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:51:44,568::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:51:44,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:51:45,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:51:45,011::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:51:45,012::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 06:51:45,014::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 06:51:45,015::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 06:51:45,016::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 06:51:45,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:51:45,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:51:45,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:51:45,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106422 s, 385 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:51:53,740::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:51:53,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828986 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:51:55,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:51:55,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916335 s, 4.5 MB/s\n'; <rc> = 0
>Thread-13884::WARNING::2013-03-17 06:52:03,152::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:52:03,155::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:52:03,775::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 06:52:03,776::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 06:52:03,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:52:03,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00129085 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:52:03,818::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00129085 s, 1.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 06:52:03,818::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 06:52:03,818::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 06:52:03,820::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 06:52:03,821::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:52:03,822::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:52:03,822::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:52:03,822::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:52:03,823::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:52:03,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:52:04,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:52:04,273::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:52:04,273::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 06:52:04,276::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 06:52:04,277::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 06:52:04,278::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 06:52:04,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:52:04,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:52:04,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:52:04,739::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174412 s, 235 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:52:05,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:52:05,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177488 s, 231 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:52:13,160::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:52:13,161::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:52:13,587::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:52:13,594::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:52:13,596::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:52:14,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:52:14,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164557 s, 249 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:52:15,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:52:15,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871114 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:52:24,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:52:24,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135638 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:52:25,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:52:25,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000708111 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:52:34,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:52:34,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819744 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:52:35,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:52:35,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000967441 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:52:44,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:52:44,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754159 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:52:45,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:52:45,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738369 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:52:54,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:52:54,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152528 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:52:55,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:52:55,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00222159 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:53:04,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:53:05,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157125 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:53:05,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:53:06,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777896 s, 5.3 MB/s\n'; <rc> = 0
>Thread-13885::WARNING::2013-03-17 06:53:13,624::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:53:13,630::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:53:15,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:53:15,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850719 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:53:16,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:53:16,081::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757596 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:53:23,633::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:53:23,634::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:53:24,161::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:53:24,168::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:53:24,169::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:53:25,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:53:25,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0236826 s, 173 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:53:26,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:53:26,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768906 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:53:35,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:53:35,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172828 s, 237 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:53:36,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:53:36,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880291 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:53:45,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:53:45,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767461 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:53:46,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:53:46,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858481 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:53:55,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:53:55,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092455 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:53:56,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:53:56,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000734829 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:54:05,260::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:54:05,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139878 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:54:06,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:54:06,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156271 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:54:15,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:54:15,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807026 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:54:16,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:54:16,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836679 s, 4.9 MB/s\n'; <rc> = 0
>Thread-13886::WARNING::2013-03-17 06:54:24,195::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:54:24,199::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:54:25,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:54:25,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869907 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:54:26,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:54:26,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840826 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:54:34,200::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 06:54:34,204::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:54:34,205::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:54:34,664::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:54:34,671::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:54:34,672::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:54:35,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:54:35,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123713 s, 331 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:54:36,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:54:36,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000947085 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:54:45,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:54:45,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155377 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:54:46,697::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:54:46,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001079 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:54:55,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:54:55,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835942 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:54:56,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:54:56,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014694 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:55:05,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:55:05,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153436 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:55:06,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:55:06,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871629 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:55:15,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:55:15,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113586 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:55:16,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:55:16,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000908212 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:55:25,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:55:25,636::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000959011 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:55:26,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:55:27,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016368 s, 2.5 MB/s\n'; <rc> = 0
>Thread-13887::WARNING::2013-03-17 06:55:34,697::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:55:34,701::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:55:35,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:55:35,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139752 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:55:37,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:55:37,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00560931 s, 730 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:55:44,703::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:55:44,704::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:55:45,126::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:55:45,138::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:55:45,140::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:55:45,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:55:45,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00932523 s, 439 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:55:47,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:55:47,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148961 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:55:55,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:55:55,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00094612 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:55:57,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:55:57,266::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076827 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:56:05,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:56:05,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832128 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:56:07,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:56:07,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139055 s, 295 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:56:15,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:56:15,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152627 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:56:17,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:56:17,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000942497 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:56:25,854::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:56:25,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821136 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:56:27,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:56:27,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000716444 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:56:35,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:56:35,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00090252 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:56:37,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:56:37,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0306137 s, 134 kB/s\n'; <rc> = 0
>Thread-13888::WARNING::2013-03-17 06:56:45,167::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:56:45,170::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:56:45,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:56:45,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133597 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:56:47,651::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 06:56:47,653::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 06:56:47,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:56:47,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00144754 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:56:47,690::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00144754 s, 1.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 06:56:47,690::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 06:56:47,690::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 06:56:47,692::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 06:56:47,693::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:56:47,694::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:56:47,694::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:56:47,694::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:56:47,695::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 06:56:47,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:56:48,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:56:48,126::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 06:56:48,127::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 06:56:48,129::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 06:56:48,130::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 06:56:48,131::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 06:56:48,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:56:48,484::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:56:48,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:56:48,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132863 s, 308 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:56:55,175::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:56:55,176::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:56:55,574::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:56:55,581::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:56:55,582::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:56:55,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:56:56,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129643 s, 316 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:56:58,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:56:58,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155573 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:57:06,022::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 06:57:06,023::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 06:57:06,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:57:06,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000832202 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:57:06,076::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000832202 s, 2.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 06:57:06,076::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 06:57:06,077::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 06:57:06,079::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 06:57:06,080::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:57:06,081::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:57:06,082::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:57:06,082::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:57:06,083::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 06:57:06,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:57:06,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:57:06,472::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:57:06,473::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 06:57:06,475::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 06:57:06,477::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 06:57:06,477::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 06:57:06,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:57:06,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:57:06,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:57:06,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118461 s, 346 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:57:08,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:57:08,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197171 s, 208 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:57:16,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:57:17,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152299 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:57:18,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:57:18,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138214 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:57:27,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:57:27,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.021867 s, 187 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:57:28,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:57:28,866::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120295 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:57:37,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:57:37,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785797 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:57:38,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:57:38,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081702 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:57:47,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:57:47,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789221 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:57:48,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:57:49,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013831 s, 3.0 MB/s\n'; <rc> = 0
>Thread-13889::WARNING::2013-03-17 06:57:55,612::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:57:55,615::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:57:57,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:57:57,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111624 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:57:59,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:57:59,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105936 s, 3.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:58:05,617::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:58:05,619::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:58:06,017::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:58:06,024::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:58:06,025::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:58:07,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:58:07,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017945 s, 228 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:58:09,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:58:09,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847836 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:58:17,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:58:17,404::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146551 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:58:19,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:58:19,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880407 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:58:27,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:58:27,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079632 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:58:29,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:58:29,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0009132 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:58:37,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:58:37,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847411 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:58:39,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:58:39,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157973 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:58:47,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:58:47,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136398 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:58:49,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:58:49,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761228 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:58:57,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:58:57,631::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015388 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:58:59,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:58:59,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166258 s, 2.5 MB/s\n'; <rc> = 0
>Thread-13890::WARNING::2013-03-17 06:59:06,052::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 06:59:06,056::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 06:59:07,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:59:07,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000956673 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:59:09,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:59:09,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166241 s, 2.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 06:59:16,058::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 06:59:16,059::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 06:59:16,494::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 06:59:16,503::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 06:59:16,504::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 06:59:17,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:59:17,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147876 s, 277 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:59:19,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:59:19,692::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149379 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:59:27,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:59:27,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081438 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:59:29,732::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:59:29,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853341 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:59:37,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:59:37,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000945023 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:59:39,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:59:39,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178981 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:59:47,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:59:47,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000965529 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:59:49,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 06:59:49,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148867 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 06:59:57,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 06:59:57,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151551 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 06:59:59,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:00:00,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142817 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:00:07,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:00:07,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143639 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:00:10,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:00:10,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152932 s, 2.7 MB/s\n'; <rc> = 0
>Thread-13891::WARNING::2013-03-17 07:00:16,534::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:00:16,539::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:00:17,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:00:17,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860904 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:00:20,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:00:20,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120453 s, 3.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:00:26,541::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 07:00:26,542::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:00:26,544::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:00:26,973::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:00:26,980::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:00:26,981::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:00:27,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:00:28,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126913 s, 323 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:00:30,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:00:30,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907212 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:00:38,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:00:38,060::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776374 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:00:40,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:00:40,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109387 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:00:48,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:00:48,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109763 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:00:50,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:00:50,424::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149772 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:00:58,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:00:58,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147006 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:01:00,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:01:00,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848808 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:01:08,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:01:08,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883916 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:01:10,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:01:10,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017056 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:01:18,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:01:18,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747212 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:01:20,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:01:20,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00231868 s, 1.8 MB/s\n'; <rc> = 0
>Thread-13892::WARNING::2013-03-17 07:01:27,010::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:01:27,016::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:01:28,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:01:28,260::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777148 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:01:30,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:01:30,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016121 s, 2.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:01:37,018::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:01:37,019::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:01:37,444::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:01:37,450::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:01:37,452::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:01:38,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:01:38,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153881 s, 266 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:01:40,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:01:40,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0396458 s, 103 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:01:48,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:01:48,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153977 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:01:50,891::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 07:01:50,892::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 07:01:50,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:01:50,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00148289 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:01:50,931::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00148289 s, 1.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 07:01:50,932::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 07:01:50,932::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 07:01:50,934::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 07:01:50,935::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:01:50,936::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:01:50,937::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:01:50,937::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:01:50,938::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:01:50,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:01:51,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:01:51,345::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:01:51,346::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 07:01:51,348::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 07:01:51,349::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 07:01:51,350::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 07:01:51,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:01:51,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:01:51,730::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:01:51,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00911736 s, 449 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:01:58,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:01:58,401::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00496736 s, 825 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:02:01,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:02:01,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820978 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:02:08,405::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 07:02:08,406::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 07:02:08,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:02:08,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00138764 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:02:08,446::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00138764 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 07:02:08,447::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 07:02:08,447::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 07:02:08,449::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 07:02:08,450::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:02:08,451::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:02:08,451::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:02:08,452::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:02:08,452::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:02:08,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:02:08,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:02:08,898::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:02:08,900::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 07:02:08,902::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 07:02:08,904::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 07:02:08,904::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 07:02:08,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:02:09,270::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:02:09,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:02:09,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194306 s, 211 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:02:11,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:02:11,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835963 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:02:19,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:02:19,372::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162501 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:02:21,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:02:21,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828916 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:02:29,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:02:29,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143538 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:02:32,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:02:32,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000972167 s, 4.2 MB/s\n'; <rc> = 0
>Thread-13893::WARNING::2013-03-17 07:02:37,477::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:02:37,481::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:02:39,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:02:39,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842394 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:02:42,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:02:42,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853052 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:02:47,483::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:02:47,484::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:02:47,915::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:02:47,921::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:02:47,923::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:02:49,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:02:49,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163365 s, 251 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:02:52,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:02:52,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170343 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:02:59,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:02:59,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132401 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:03:02,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:03:02,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000672072 s, 6.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:03:09,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:03:09,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837862 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:03:12,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:03:12,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879662 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:03:19,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:03:19,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158619 s, 258 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:03:22,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:03:22,448::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140074 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:03:29,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:03:29,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0030561 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:03:32,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:03:32,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923021 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:03:39,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:03:39,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848098 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:03:42,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:03:42,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139441 s, 2.9 MB/s\n'; <rc> = 0
>Thread-13894::WARNING::2013-03-17 07:03:47,948::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:03:47,952::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:03:49,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:03:49,777::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789996 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:03:52,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:03:52,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102064 s, 4.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:03:57,954::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:03:57,955::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:03:58,386::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:03:58,393::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:03:58,394::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:03:59,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:03:59,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137123 s, 299 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:04:02,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:04:02,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851147 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:04:09,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:04:09,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161382 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:04:12,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:04:12,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140377 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:04:19,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:04:19,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812504 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:04:22,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:04:22,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00093991 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:04:29,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:04:29,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014179 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:04:32,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:04:32,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824764 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:04:39,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:04:40,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149065 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:04:43,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:04:43,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139335 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:04:50,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:04:50,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763561 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:04:53,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:04:53,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143152 s, 2.9 MB/s\n'; <rc> = 0
>Thread-13895::WARNING::2013-03-17 07:04:58,426::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:04:58,431::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:05:00,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:05:00,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915061 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:05:03,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:05:03,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756785 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:05:08,434::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:05:08,435::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:05:08,857::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:05:08,864::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:05:08,865::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:05:10,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:05:10,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0207612 s, 197 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:05:13,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:05:13,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150883 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:05:20,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:05:20,195::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000942753 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:05:23,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:05:23,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925135 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:05:30,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:05:30,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140118 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:05:33,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:05:33,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128233 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:05:40,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:05:40,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152295 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:05:43,481::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:05:43,512::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861005 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:05:50,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:05:50,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000977394 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:05:53,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:05:53,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134458 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:06:00,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:06:00,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142123 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:06:03,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:06:03,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151165 s, 2.7 MB/s\n'; <rc> = 0
>Thread-13896::WARNING::2013-03-17 07:06:08,900::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:06:08,903::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:06:10,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:06:10,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00180508 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:06:13,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:06:13,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000705855 s, 5.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:06:18,904::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 07:06:18,906::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:06:18,907::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:06:19,340::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:06:19,347::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:06:19,349::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:06:20,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:06:20,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121085 s, 338 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:06:23,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:06:23,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871828 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:06:30,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:06:30,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080172 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:06:33,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:06:33,907::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151629 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:06:40,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:06:40,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869589 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:06:43,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:06:44,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140137 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:06:50,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:06:50,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151881 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:06:54,053::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 07:06:54,054::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 07:06:54,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:06:54,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00107006 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:06:54,097::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00107006 s, 1.9 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 07:06:54,098::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 07:06:54,098::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 07:06:54,099::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 07:06:54,100::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:06:54,101::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:06:54,102::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:06:54,102::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:06:54,102::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:06:54,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:06:54,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:06:54,553::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:06:54,554::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 07:06:54,557::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 07:06:54,558::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 07:06:54,558::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 07:06:54,560::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:06:54,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:06:54,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:06:54,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00727541 s, 563 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:07:00,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:07:00,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811616 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:07:05,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:07:05,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173249 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:07:10,654::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 07:07:10,655::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 07:07:10,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:07:10,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00140941 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:07:10,698::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00140941 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 07:07:10,698::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 07:07:10,698::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 07:07:10,700::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 07:07:10,701::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:07:10,701::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:07:10,702::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:07:10,702::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:07:10,703::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:07:10,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:07:11,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:07:11,150::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:07:11,151::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 07:07:11,155::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 07:07:11,156::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 07:07:11,156::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 07:07:11,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:07:11,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:07:11,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:07:11,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172979 s, 237 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:07:15,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:07:15,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125056 s, 3.3 MB/s\n'; <rc> = 0
>Thread-13897::WARNING::2013-03-17 07:07:19,375::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:07:19,381::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:07:21,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:07:21,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142005 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:07:25,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:07:25,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852397 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:07:29,384::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:07:29,385::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:07:29,813::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:07:29,820::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:07:29,821::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:07:31,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:07:31,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00296826 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:07:35,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:07:35,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806288 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:07:41,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:07:41,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010288 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:07:45,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:07:45,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814249 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:07:51,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:07:51,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153232 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:07:55,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:07:55,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168917 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:08:01,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:08:01,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172755 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:08:05,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:08:05,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771192 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:08:11,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:08:11,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014208 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:08:15,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:08:15,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850828 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:08:21,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:08:21,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811997 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:08:25,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:08:25,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148462 s, 2.8 MB/s\n'; <rc> = 0
>Thread-13898::WARNING::2013-03-17 07:08:29,847::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:08:29,850::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:08:31,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:08:32,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109618 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:08:35,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:08:35,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00997285 s, 411 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:08:39,853::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:08:39,854::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:08:40,280::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:08:40,287::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:08:40,288::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:08:42,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:08:42,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139792 s, 293 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:08:45,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:08:45,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764614 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:08:52,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:08:52,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913653 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:08:55,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:08:55,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104974 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:09:02,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:09:02,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014028 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:09:05,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:09:05,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147227 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:09:12,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:09:12,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142753 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:09:16,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:09:16,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802319 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:09:22,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:09:22,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00070924 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:09:26,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:09:26,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151968 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:09:32,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:09:32,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886534 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:09:36,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:09:36,191::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757225 s, 5.4 MB/s\n'; <rc> = 0
>Thread-13899::WARNING::2013-03-17 07:09:40,314::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:09:40,319::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:09:42,350::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:09:42,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012178 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:09:46,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:09:46,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00096797 s, 4.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:09:50,323::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:09:50,324::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:09:50,794::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:09:50,801::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:09:50,802::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:09:52,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:09:52,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129048 s, 317 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:09:56,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:09:56,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00821144 s, 499 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:10:02,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:10:02,521::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138242 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:10:06,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:10:06,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000973137 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:10:12,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:10:12,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872679 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:10:16,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:10:16,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0025313 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:10:22,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:10:22,627::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142668 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:10:26,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:10:26,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144806 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:10:32,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:10:32,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0332076 s, 123 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:10:36,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:10:36,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012356 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:10:42,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:10:42,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136338 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:10:46,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:10:46,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000736179 s, 5.6 MB/s\n'; <rc> = 0
>Thread-13900::WARNING::2013-03-17 07:10:50,832::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:10:50,841::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:10:52,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:10:52,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854979 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:10:56,781::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:10:56,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765478 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:11:00,844::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:11:00,845::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:11:01,250::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:11:01,257::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:11:01,258::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:11:02,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:11:02,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137676 s, 298 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:11:06,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:11:06,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000992327 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:11:12,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:11:12,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019856 s, 206 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:11:16,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:11:16,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168562 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:11:22,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:11:22,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000975376 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:11:27,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:11:27,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0219978 s, 186 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:11:32,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:11:32,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158048 s, 259 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:11:37,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:11:37,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817826 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:11:43,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:11:43,046::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139603 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:11:47,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:11:47,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798142 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:11:53,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:11:53,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144007 s, 284 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:11:57,294::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 07:11:57,304::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 07:11:57,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:11:57,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000827203 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:11:57,339::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000827203 s, 2.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 07:11:57,340::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 07:11:57,340::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 07:11:57,343::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 07:11:57,344::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:11:57,345::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:11:57,345::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:11:57,345::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:11:57,346::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:11:57,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:11:57,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:11:57,805::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:11:57,806::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 07:11:57,808::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 07:11:57,810::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 07:11:57,810::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 07:11:57,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:11:58,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:11:58,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:11:58,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00450186 s, 910 kB/s\n'; <rc> = 0
>Thread-13901::WARNING::2013-03-17 07:12:01,291::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:12:01,294::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:12:03,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:12:03,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0355499 s, 115 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:12:08,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:12:08,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0284867 s, 144 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:12:11,296::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 07:12:11,297::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:12:11,298::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:12:11,776::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:12:11,783::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:12:11,784::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:12:13,179::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 07:12:13,182::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 07:12:13,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:12:13,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0135269 s, 151 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:12:13,223::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0135269 s, 151 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 07:12:13,224::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 07:12:13,224::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 07:12:13,227::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 07:12:13,228::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:12:13,229::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:12:13,230::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:12:13,230::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:12:13,230::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:12:13,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:12:13,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:12:13,650::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:12:13,651::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 07:12:13,653::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 07:12:13,655::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 07:12:13,655::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 07:12:13,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:12:14,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:12:14,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:12:14,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000968356 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:12:18,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:12:18,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0238875 s, 171 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:12:24,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:12:24,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138259 s, 296 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:12:28,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:12:28,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082756 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:12:34,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:12:34,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130099 s, 315 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:12:38,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:12:38,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080458 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:12:44,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:12:44,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897881 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:12:48,603::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:12:48,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162239 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:12:54,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:12:54,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137761 s, 297 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:12:58,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:12:58,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771542 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:13:04,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:13:04,297::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149078 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:13:08,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:13:08,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173401 s, 2.4 MB/s\n'; <rc> = 0
>Thread-13902::WARNING::2013-03-17 07:13:11,809::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:13:11,813::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:13:14,305::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:13:14,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125911 s, 325 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:13:18,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:13:18,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851448 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:13:21,815::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:13:21,816::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:13:22,210::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:13:22,216::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:13:22,218::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:13:24,352::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:13:24,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017748 s, 231 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:13:28,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:13:28,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773705 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:13:34,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:13:34,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161139 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:13:38,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:13:39,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076184 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:13:44,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:13:44,517::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172725 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:13:49,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:13:49,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150369 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:13:54,522::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:13:54,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011335 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:13:59,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:13:59,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0007165 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:14:04,556::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:14:04,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167675 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:14:09,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:14:09,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00207756 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:14:14,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:14:14,649::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010598 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:14:19,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:14:19,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00300668 s, 1.4 MB/s\n'; <rc> = 0
>Thread-13903::WARNING::2013-03-17 07:14:22,245::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:14:22,249::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:14:24,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:14:24,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781451 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:14:29,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:14:29,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00522355 s, 784 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:14:32,253::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:14:32,254::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:14:32,708::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:14:32,715::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:14:32,716::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:14:34,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:14:34,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00711937 s, 575 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:14:39,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:14:39,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802597 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:14:44,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:14:44,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136542 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:14:49,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:14:49,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839068 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:14:54,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:14:54,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016688 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:14:59,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:14:59,617::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882452 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:15:04,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:15:04,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169868 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:15:09,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:15:09,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784256 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:15:14,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:15:14,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0024497 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:15:19,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:15:19,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141511 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:15:24,910::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:15:24,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769864 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:15:29,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:15:29,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853524 s, 4.8 MB/s\n'; <rc> = 0
>Thread-13904::WARNING::2013-03-17 07:15:32,746::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:15:32,751::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:15:34,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:15:34,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00428384 s, 956 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:15:39,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:15:39,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855556 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:15:42,754::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:15:42,756::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:15:43,203::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:15:43,210::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:15:43,211::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:15:44,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:15:45,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00684611 s, 598 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:15:49,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:15:50,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160649 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:15:55,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:15:55,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810284 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:16:00,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:16:00,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00189787 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:16:05,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:16:05,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869244 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:16:10,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:16:10,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00826973 s, 495 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:16:15,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:16:15,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890629 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:16:20,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:16:20,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806766 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:16:25,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:16:25,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000735321 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:16:30,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:16:30,350::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014065 s, 291 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:16:35,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:16:35,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00206856 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:16:40,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:16:40,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112881 s, 3.6 MB/s\n'; <rc> = 0
>Thread-13905::WARNING::2013-03-17 07:16:43,237::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:16:43,240::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:16:45,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:16:45,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137257 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:16:50,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:16:50,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00189639 s, 2.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:16:53,252::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:16:53,253::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:16:53,685::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:16:53,692::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:16:53,694::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:16:55,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:16:55,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177431 s, 231 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:17:00,580::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 07:17:00,581::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 07:17:00,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:17:00,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000787069 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:17:00,615::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000787069 s, 2.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 07:17:00,616::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 07:17:00,616::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 07:17:00,618::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 07:17:00,619::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:17:00,619::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:17:00,620::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:17:00,620::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:17:00,621::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:17:00,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:17:01,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:17:01,048::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:17:01,049::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 07:17:01,051::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 07:17:01,053::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 07:17:01,053::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 07:17:01,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:17:01,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:17:01,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:17:01,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886076 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:17:05,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:17:05,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176362 s, 232 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:17:11,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:17:11,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077805 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:17:15,415::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 07:17:15,416::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 07:17:15,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:17:15,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000819258 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:17:15,445::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000819258 s, 2.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 07:17:15,446::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 07:17:15,446::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 07:17:15,448::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 07:17:15,453::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:17:15,454::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:17:15,454::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:17:15,454::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:17:15,455::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:17:15,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:17:15,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:17:15,883::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:17:15,884::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 07:17:15,887::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 07:17:15,888::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 07:17:15,888::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 07:17:15,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:17:16,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:17:16,281::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:17:16,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134003 s, 306 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:17:21,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:17:21,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144529 s, 283 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:17:26,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:17:26,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082479 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:17:31,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:17:31,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076634 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:17:36,365::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:17:36,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088086 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:17:41,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:17:41,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143564 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:17:46,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:17:46,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140611 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:17:51,792::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:17:51,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802305 s, 5.1 MB/s\n'; <rc> = 0
>Thread-13906::WARNING::2013-03-17 07:17:53,721::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:17:53,725::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:17:56,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:17:56,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795093 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:18:02,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:18:02,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00298475 s, 1.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:18:03,726::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 07:18:03,728::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:18:03,729::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:18:04,143::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:18:04,149::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:18:04,151::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:18:06,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:18:06,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105506 s, 388 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:18:12,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:18:12,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00584883 s, 700 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:18:16,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:18:16,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850809 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:18:22,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:18:22,297::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894784 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:18:26,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:18:26,611::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154319 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:18:32,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:18:32,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110556 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:18:36,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:18:36,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858353 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:18:42,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:18:42,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175873 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:18:46,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:18:46,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170365 s, 240 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:18:52,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:18:52,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144377 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:18:56,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:18:56,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144965 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:19:02,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:19:02,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911529 s, 4.5 MB/s\n'; <rc> = 0
>Thread-13907::WARNING::2013-03-17 07:19:04,177::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:19:04,181::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:19:06,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:19:06,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01202 s, 341 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:19:12,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:19:12,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00281981 s, 1.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:19:14,183::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:19:14,185::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:19:14,590::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:19:14,596::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:19:14,598::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:19:16,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:19:16,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142892 s, 287 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:19:22,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:19:22,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137392 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:19:26,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:19:26,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00952011 s, 430 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:19:33,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:19:33,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798278 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:19:36,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:19:36,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106371 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:19:43,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:19:43,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000687029 s, 6.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:19:46,950::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:19:46,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119413 s, 343 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:19:53,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:19:53,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881434 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:19:56,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:19:57,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867317 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:20:03,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:20:03,260::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154436 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:20:07,034::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:20:07,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181673 s, 225 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:20:13,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:20:13,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826293 s, 5.0 MB/s\n'; <rc> = 0
>Thread-13908::WARNING::2013-03-17 07:20:14,625::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:20:14,628::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:20:17,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:20:17,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805763 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:20:23,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:20:23,517::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100285 s, 4.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:20:24,631::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:20:24,632::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:20:25,067::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:20:25,073::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:20:25,075::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:20:27,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:20:27,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114019 s, 359 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:20:33,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:20:33,598::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000739209 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:20:37,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:20:37,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135252 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:20:43,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:20:43,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125773 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:20:47,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:20:47,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127474 s, 321 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:20:53,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:20:53,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157753 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:20:57,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:20:57,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834282 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:21:03,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:21:03,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773783 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:21:07,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:21:07,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128887 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:21:13,871::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:21:13,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075047 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:21:17,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:21:17,399::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925426 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:21:23,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:21:23,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128553 s, 3.2 MB/s\n'; <rc> = 0
>Thread-13909::WARNING::2013-03-17 07:21:25,101::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:21:25,105::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:21:27,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:21:27,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168463 s, 243 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:21:34,034::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:21:34,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00883633 s, 464 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:21:35,107::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:21:35,109::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:21:35,531::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:21:35,538::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:21:35,539::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:21:37,455::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:21:37,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132931 s, 308 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:21:44,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:21:44,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120561 s, 340 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:21:47,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:21:47,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149443 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:21:54,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:21:54,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803581 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:21:57,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:21:57,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159723 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:22:04,302::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 07:22:04,303::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 07:22:04,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:22:04,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00152728 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:22:04,350::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00152728 s, 1.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 07:22:04,350::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 07:22:04,350::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 07:22:04,352::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 07:22:04,353::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:22:04,354::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:22:04,354::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:22:04,355::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:22:04,355::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:22:04,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:22:04,786::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:22:04,807::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:22:04,816::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 07:22:04,830::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 07:22:04,832::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 07:22:04,833::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 07:22:04,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:22:05,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:22:05,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:22:05,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104422 s, 392 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:22:07,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:22:07,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199878 s, 205 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:22:15,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:22:15,331::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176712 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:22:17,650::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 07:22:17,652::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 07:22:17,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:22:17,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00132413 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:22:17,691::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00132413 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 07:22:17,694::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 07:22:17,695::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 07:22:17,696::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 07:22:17,697::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:22:17,698::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:22:17,698::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:22:17,698::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:22:17,699::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:22:17,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:22:18,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:22:18,147::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:22:18,148::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 07:22:18,150::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 07:22:18,152::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 07:22:18,152::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 07:22:18,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:22:18,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:22:18,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:22:18,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00534152 s, 767 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:22:25,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:22:25,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797227 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:22:28,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:22:28,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115234 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:22:35,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:22:35,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767441 s, 5.3 MB/s\n'; <rc> = 0
>Thread-13910::WARNING::2013-03-17 07:22:35,571::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:22:35,586::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:22:38,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:22:38,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0028938 s, 1.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:22:45,589::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:22:45,591::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:22:45,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:22:45,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142697 s, 287 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:22:46,141::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:22:46,147::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:22:46,148::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:22:48,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:22:48,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010316 s, 397 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:22:55,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:22:55,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000729837 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:22:58,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:22:58,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118697 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:23:05,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:23:05,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897893 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:23:08,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:23:08,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00096046 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:23:15,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:23:15,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000692898 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:23:18,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:23:18,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842518 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:23:25,933::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:23:25,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00090084 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:23:28,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:23:28,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815241 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:23:36,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:23:36,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151668 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:23:38,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:23:38,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136783 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:23:46,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:23:46,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0048428 s, 846 kB/s\n'; <rc> = 0
>Thread-13911::WARNING::2013-03-17 07:23:46,180::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:23:46,191::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:23:48,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:23:48,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117632 s, 348 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:23:56,195::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 07:23:56,197::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:23:56,198::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:23:56,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:23:56,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141766 s, 289 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:23:56,716::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:23:56,721::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:23:56,722::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:23:58,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:23:58,963::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00620608 s, 660 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:24:06,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:24:06,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141203 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:24:08,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:24:09,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000949345 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:24:16,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:24:16,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756846 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:24:19,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:24:19,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137967 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:24:26,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:24:26,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827756 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:24:29,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:24:29,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011087 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:24:36,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:24:36,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142948 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:24:39,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:24:39,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000957021 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:24:46,681::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:24:46,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171515 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:24:49,117::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:24:49,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143741 s, 2.8 MB/s\n'; <rc> = 0
>Thread-13912::WARNING::2013-03-17 07:24:56,749::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:24:56,753::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 07:24:56,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:24:56,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853914 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:24:59,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:24:59,191::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813243 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:25:06,755::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:25:06,757::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:25:06,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:25:06,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164825 s, 2.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:25:07,238::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:25:07,244::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:25:07,245::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:25:09,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:25:09,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136084 s, 301 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:25:16,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:25:16,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779554 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:25:19,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:25:19,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000979681 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:25:27,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:25:27,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892792 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:25:29,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:25:29,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808492 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:25:37,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:25:37,123::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172806 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:25:39,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:25:39,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162248 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:25:47,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:25:47,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836272 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:25:49,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:25:49,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143941 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:25:57,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:25:57,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880436 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:25:59,415::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:25:59,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014632 s, 2.8 MB/s\n'; <rc> = 0
>Thread-13913::WARNING::2013-03-17 07:26:07,272::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:26:07,279::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 07:26:07,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:26:07,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000927836 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:26:09,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:26:09,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000985187 s, 4.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:26:17,282::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:26:17,284::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:26:17,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:26:17,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155217 s, 264 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:26:17,817::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:26:17,822::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:26:17,823::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:26:19,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:26:19,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116389 s, 352 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:26:27,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:26:27,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816033 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:26:29,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:26:29,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835121 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:26:37,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:26:37,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00233324 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:26:39,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:26:39,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012927 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:26:47,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:26:47,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773363 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:26:49,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:26:49,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820717 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:26:57,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:26:57,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891203 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:26:59,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:26:59,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145801 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:27:07,813::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 07:27:07,814::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 07:27:07,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:27:07,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00132167 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:27:07,852::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00132167 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 07:27:07,853::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 07:27:07,853::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 07:27:07,855::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 07:27:07,859::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:27:07,860::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:27:07,861::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:27:07,861::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:27:07,861::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:27:07,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:27:08,299::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:27:08,306::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:27:08,309::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 07:27:08,311::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 07:27:08,312::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 07:27:08,313::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 07:27:08,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:27:08,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:27:08,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:27:08,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00641283 s, 639 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:27:09,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:27:09,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112061 s, 366 kB/s\n'; <rc> = 0
>Thread-13914::WARNING::2013-03-17 07:27:17,849::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:27:17,853::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 07:27:18,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:27:18,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799795 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:27:19,790::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 07:27:19,792::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 07:27:19,792::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:27:19,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00962026 s, 213 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:27:19,856::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00962026 s, 213 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 07:27:19,856::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 07:27:19,856::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 07:27:19,858::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 07:27:19,859::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:27:19,860::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:27:19,860::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:27:19,861::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:27:19,861::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:27:19,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:27:20,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:27:20,265::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:27:20,266::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 07:27:20,269::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 07:27:20,270::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 07:27:20,270::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 07:27:20,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:27:20,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:27:20,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:27:20,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173478 s, 2.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:27:27,855::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:27:27,856::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:27:28,266::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:27:28,272::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:27:28,274::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:27:28,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:27:28,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144888 s, 283 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:27:30,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:27:30,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129999 s, 315 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:27:38,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:27:38,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000725995 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:27:40,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:27:40,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827935 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:27:48,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:27:49,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000701058 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:27:50,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:27:50,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143238 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:27:59,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:27:59,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799354 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:28:00,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:28:00,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151684 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:28:09,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:28:09,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135045 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:28:10,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:28:10,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805259 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:28:19,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:28:19,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171777 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:28:20,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:28:20,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00233664 s, 1.8 MB/s\n'; <rc> = 0
>Thread-13915::WARNING::2013-03-17 07:28:28,306::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:28:28,309::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 07:28:29,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:28:29,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819028 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:28:30,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:28:30,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781336 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:28:38,311::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:28:38,313::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:28:38,742::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:28:38,749::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:28:38,750::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:28:39,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:28:39,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105942 s, 387 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:28:40,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:28:41,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00885925 s, 462 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:28:49,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:28:49,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829112 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:28:51,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:28:51,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127055 s, 322 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:28:59,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:28:59,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153594 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:29:01,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:29:01,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0086487 s, 474 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:29:09,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:29:09,636::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897214 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:29:11,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:29:11,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014441 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:29:19,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:29:19,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924667 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:29:21,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:29:21,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141619 s, 289 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:29:29,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:29:29,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146012 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:29:31,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:29:31,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109418 s, 3.7 MB/s\n'; <rc> = 0
>Thread-13916::WARNING::2013-03-17 07:29:38,776::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:29:38,780::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 07:29:39,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:29:39,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853281 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:29:41,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:29:41,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143398 s, 286 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:29:48,782::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 07:29:48,784::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:29:48,785::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:29:49,178::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:29:49,185::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:29:49,186::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:29:49,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:29:49,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143442 s, 286 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:29:51,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:29:51,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115666 s, 354 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:29:59,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:30:00,010::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850989 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:30:01,365::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:30:01,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153963 s, 266 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:30:10,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:30:10,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143552 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:30:11,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:30:11,552::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158125 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:30:20,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:30:20,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161595 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:30:21,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:30:21,602::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150987 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:30:30,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:30:30,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014297 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:30:31,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:30:31,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015923 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:30:40,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:30:40,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864493 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:30:41,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:30:41,698::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153654 s, 2.7 MB/s\n'; <rc> = 0
>Thread-13917::WARNING::2013-03-17 07:30:49,213::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:30:49,217::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 07:30:50,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:30:50,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919285 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:30:51,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:30:51,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748295 s, 5.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:30:59,221::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:30:59,223::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:30:59,679::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:30:59,686::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:30:59,687::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:31:00,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:31:00,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164021 s, 250 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:31:01,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:31:01,793::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179221 s, 229 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:31:10,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:31:10,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149003 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:31:11,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:31:11,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120946 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:31:20,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:31:20,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131501 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:31:21,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:31:21,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0238179 s, 172 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:31:30,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:31:30,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0019719 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:31:31,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:31:31,944::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008703 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:31:40,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:31:40,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154573 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:31:41,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:31:41,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00186227 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:31:50,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:31:50,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757925 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:31:51,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:31:52,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013298 s, 3.1 MB/s\n'; <rc> = 0
>Thread-13918::WARNING::2013-03-17 07:31:59,717::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:31:59,731::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 07:32:00,933::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:32:00,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000994598 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:32:02,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:32:02,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175182 s, 2.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:32:09,735::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:32:09,737::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:32:10,189::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:32:10,196::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:32:10,197::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:32:11,004::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 07:32:11,007::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 07:32:11,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:32:11,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0149339 s, 137 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:32:11,052::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0149339 s, 137 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 07:32:11,053::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 07:32:11,053::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 07:32:11,055::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 07:32:11,056::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:32:11,057::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:32:11,057::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:32:11,057::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:32:11,058::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:32:11,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:32:11,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:32:11,437::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:32:11,438::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 07:32:11,441::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 07:32:11,442::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 07:32:11,443::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 07:32:11,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:32:11,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:32:11,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:32:11,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190459 s, 215 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:32:12,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:32:12,123::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00490294 s, 835 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:32:21,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:32:21,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075812 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:32:22,128::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 07:32:22,130::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 07:32:22,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:32:22,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000819783 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:32:22,158::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000819783 s, 2.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 07:32:22,159::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 07:32:22,159::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 07:32:22,161::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 07:32:22,162::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:32:22,162::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:32:22,163::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:32:22,163::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:32:22,164::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:32:22,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:32:22,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:32:22,570::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:32:22,571::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 07:32:22,574::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 07:32:22,575::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 07:32:22,575::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 07:32:22,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:32:22,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:32:22,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:32:22,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00282268 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:32:31,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:32:31,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880359 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:32:32,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:32:33,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101367 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:32:42,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:32:42,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138493 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:32:43,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:32:43,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000997137 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:32:52,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:32:52,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133298 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:32:53,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:32:53,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136814 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:33:02,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:33:02,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00228123 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:33:03,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:33:03,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906442 s, 4.5 MB/s\n'; <rc> = 0
>Thread-13919::WARNING::2013-03-17 07:33:10,225::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:33:10,228::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 07:33:12,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:33:12,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852892 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:33:13,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:33:13,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085134 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:33:20,231::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:33:20,232::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:33:20,717::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:33:20,724::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:33:20,725::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:33:22,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:33:22,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189495 s, 216 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:33:23,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:33:23,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135925 s, 301 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:33:32,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:33:32,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087762 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:33:33,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:33:33,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866022 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:33:42,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:33:42,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856434 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:33:43,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:33:43,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00299017 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:33:52,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:33:52,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155041 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:33:53,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:33:53,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00709219 s, 578 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:34:02,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:34:02,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788558 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:34:03,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:34:03,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161593 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:34:12,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:34:12,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134853 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:34:13,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:34:13,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00297861 s, 1.4 MB/s\n'; <rc> = 0
>Thread-13920::WARNING::2013-03-17 07:34:20,753::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:34:20,755::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 07:34:22,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:34:22,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794332 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:34:23,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:34:23,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805023 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:34:30,758::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:34:30,759::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:34:31,158::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:34:31,165::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:34:31,166::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:34:32,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:34:32,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899711 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:34:33,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:34:33,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167526 s, 244 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:34:42,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:34:42,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170194 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:34:43,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:34:43,569::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153655 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:34:53,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:34:53,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864957 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:34:53,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:34:53,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148363 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:35:03,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:35:03,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0018629 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:35:03,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:35:03,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00228845 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:35:13,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:35:13,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156678 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:35:13,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:35:13,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847254 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:35:23,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:35:23,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111165 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:35:23,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:35:23,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015461 s, 2.6 MB/s\n'; <rc> = 0
>Thread-13921::WARNING::2013-03-17 07:35:31,194::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:35:31,198::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 07:35:33,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:35:33,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00265111 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:35:33,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:35:33,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925854 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:35:41,200::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 07:35:41,201::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:35:41,202::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:35:41,617::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:35:41,623::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:35:41,625::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:35:43,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:35:43,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747545 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:35:43,781::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:35:43,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127637 s, 321 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:35:53,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:35:53,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128531 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:35:53,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:35:53,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130351 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:36:03,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:36:03,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875239 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:36:03,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:36:03,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809543 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:36:13,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:36:13,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087573 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:36:13,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:36:13,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00246483 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:36:23,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:36:23,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794862 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:36:23,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:36:24,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139237 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:36:33,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:36:33,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00209499 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:36:34,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:36:34,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159731 s, 2.6 MB/s\n'; <rc> = 0
>Thread-13922::WARNING::2013-03-17 07:36:41,649::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:36:41,653::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 07:36:43,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:36:43,928::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000922736 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:36:44,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:36:44,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140888 s, 291 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:36:51,668::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:36:51,670::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:36:52,067::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:36:52,074::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:36:52,075::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:36:53,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:36:53,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794267 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:36:54,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:36:54,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154068 s, 266 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:37:04,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:37:04,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00941402 s, 435 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:37:04,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:37:04,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886208 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:37:14,115::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 07:37:14,116::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 07:37:14,117::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:37:14,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000926677 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:37:14,156::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000926677 s, 2.2 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 07:37:14,156::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 07:37:14,157::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 07:37:14,159::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 07:37:14,160::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:37:14,161::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:37:14,162::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:37:14,162::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:37:14,162::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:37:14,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:37:14,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:37:14,270::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194312 s, 211 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:37:14,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:37:14,609::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:37:14,610::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 07:37:14,612::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 07:37:14,613::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 07:37:14,613::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 07:37:14,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:37:15,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:37:15,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:37:15,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153967 s, 266 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:37:24,280::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 07:37:24,282::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 07:37:24,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:37:24,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0099744 s, 205 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:37:24,326::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0099744 s, 205 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 07:37:24,326::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 07:37:24,326::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 07:37:24,328::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 07:37:24,330::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:37:24,331::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:37:24,331::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:37:24,332::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:37:24,332::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:37:24,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:37:24,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:37:24,745::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:37:24,746::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 07:37:24,748::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 07:37:24,750::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 07:37:24,750::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 07:37:24,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:37:25,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:37:25,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:37:25,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:37:25,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101264 s, 404 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:37:25,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174362 s, 235 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:37:35,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:37:35,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:37:35,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890315 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:37:35,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117132 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:37:45,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:37:45,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:37:45,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085576 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:37:45,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118452 s, 3.5 MB/s\n'; <rc> = 0
>Thread-13923::WARNING::2013-03-17 07:37:52,119::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:37:52,122::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:37:55,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:37:55,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821257 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:37:55,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:37:55,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087353 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:38:02,124::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:38:02,126::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:38:02,519::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:38:02,526::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:38:02,527::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:38:05,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:38:05,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126087 s, 325 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:38:05,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:38:05,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846276 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:38:15,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:38:15,561::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165493 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:38:15,625::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:38:15,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163707 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:38:25,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:38:25,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179193 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:38:25,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:38:25,739::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000736229 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:38:35,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:38:35,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137578 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:38:35,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:38:35,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162565 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:38:45,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:38:45,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139548 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:38:45,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:38:45,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148669 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:38:55,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:38:55,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153364 s, 267 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:38:55,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:38:55,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000969317 s, 4.2 MB/s\n'; <rc> = 0
>Thread-13924::WARNING::2013-03-17 07:39:02,555::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:39:02,558::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:39:05,760::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:39:05,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107371 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:39:06,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:39:06,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100647 s, 4.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:39:12,561::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:39:12,563::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:39:12,981::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:39:12,988::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:39:12,990::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:39:15,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:39:15,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156714 s, 261 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:39:16,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:39:16,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794683 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:39:25,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:39:25,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127313 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:39:26,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:39:26,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792829 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:39:35,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:39:35,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168282 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:39:36,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:39:36,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158599 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:39:45,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:39:45,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169918 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:39:46,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:39:46,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155591 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:39:55,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:39:56,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127024 s, 322 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:39:56,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:39:56,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142765 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:40:06,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:40:06,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898887 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:40:06,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:40:06,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823693 s, 5.0 MB/s\n'; <rc> = 0
>Thread-13925::WARNING::2013-03-17 07:40:13,015::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:40:13,018::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:40:16,060::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:40:16,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151095 s, 271 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:40:16,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:40:16,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740624 s, 5.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:40:23,021::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:40:23,022::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:40:23,455::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:40:23,462::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:40:23,464::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:40:26,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:40:26,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121553 s, 337 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:40:26,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:40:26,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164431 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:40:36,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:40:36,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924704 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:40:36,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:40:36,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169103 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:40:46,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:40:46,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879661 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:40:46,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:40:46,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00258986 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:40:56,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:40:56,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171803 s, 238 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:40:56,899::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:40:56,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738937 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:41:06,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:41:06,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134491 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:41:06,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:41:07,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147026 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:41:16,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:41:16,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00202257 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:41:17,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:41:17,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133953 s, 3.1 MB/s\n'; <rc> = 0
>Thread-13926::WARNING::2013-03-17 07:41:23,489::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:41:23,493::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:41:26,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:41:26,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780527 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:41:27,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:41:27,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893479 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:41:33,494::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 07:41:33,496::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:41:33,497::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:41:33,932::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:41:33,939::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:41:33,941::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:41:36,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:41:36,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159221 s, 257 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:41:37,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:41:37,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873541 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:41:46,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:41:46,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000992703 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:41:47,271::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:41:47,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000926451 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:41:56,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:41:56,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811297 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:41:57,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:41:57,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179052 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:42:06,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:42:06,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156286 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:42:07,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:42:07,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879109 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:42:16,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:42:16,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140437 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:42:17,485::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 07:42:17,487::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 07:42:17,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:42:17,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000767976 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:42:17,522::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000767976 s, 2.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 07:42:17,522::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 07:42:17,523::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 07:42:17,524::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 07:42:17,525::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:42:17,526::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:42:17,526::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:42:17,527::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:42:17,527::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:42:17,528::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:42:17,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:42:17,956::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:42:17,957::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 07:42:17,959::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 07:42:17,960::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 07:42:17,961::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 07:42:17,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:42:18,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:42:18,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:42:18,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178963 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:42:26,623::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 07:42:26,625::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 07:42:26,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:42:26,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00126161 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:42:26,667::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00126161 s, 1.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 07:42:26,667::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 07:42:26,668::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 07:42:26,669::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 07:42:26,670::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:42:26,671::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:42:26,671::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:42:26,671::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:42:26,673::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:42:26,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:42:27,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:42:27,109::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:42:27,110::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 07:42:27,112::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 07:42:27,113::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 07:42:27,114::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 07:42:27,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:42:27,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:42:27,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:42:27,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0157203 s, 261 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:42:28,405::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:42:28,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182448 s, 225 kB/s\n'; <rc> = 0
>Thread-13927::WARNING::2013-03-17 07:42:33,983::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:42:33,986::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:42:37,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:42:37,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177189 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:42:38,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:42:38,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809524 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:42:43,989::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:42:43,990::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:42:44,438::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:42:44,445::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:42:44,447::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:42:47,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:42:47,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191049 s, 214 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:42:48,571::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:42:48,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887629 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:42:57,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:42:57,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834055 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:42:58,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:42:58,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148795 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:43:07,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:43:07,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827079 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:43:08,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:43:08,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741665 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:43:17,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:43:17,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149215 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:43:18,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:43:18,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148223 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:43:27,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:43:27,853::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151495 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:43:28,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:43:28,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852966 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:43:37,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:43:37,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00095648 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:43:38,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:43:39,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115619 s, 3.5 MB/s\n'; <rc> = 0
>Thread-13928::WARNING::2013-03-17 07:43:44,484::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:43:44,490::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:43:47,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:43:47,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916901 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:43:49,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:43:49,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136854 s, 3.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:43:54,494::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:43:54,495::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:43:54,913::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:43:54,920::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:43:54,921::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:43:57,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:43:57,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111499 s, 367 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:43:59,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:43:59,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848553 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:44:07,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:44:08,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106281 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:44:09,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:44:09,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814683 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:44:18,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:44:18,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000734394 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:44:19,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:44:19,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154381 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:44:28,045::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:44:28,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864444 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:44:29,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:44:29,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147068 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:44:38,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:44:38,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142315 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:44:39,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:44:39,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155801 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:44:48,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:44:48,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145252 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:44:49,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:44:49,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119622 s, 3.4 MB/s\n'; <rc> = 0
>Thread-13929::WARNING::2013-03-17 07:44:54,952::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:44:54,958::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:44:58,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:44:58,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0022482 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:44:59,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:44:59,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781754 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:45:04,960::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:45:04,961::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:45:05,371::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:45:05,378::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:45:05,380::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:45:08,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:45:08,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016182 s, 253 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:45:09,697::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:45:09,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867279 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:45:18,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:45:18,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107099 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:45:19,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:45:19,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743548 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:45:28,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:45:28,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000742097 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:45:29,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:45:29,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838137 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:45:38,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:45:38,562::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852179 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:45:39,899::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:45:39,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145915 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:45:48,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:45:48,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077061 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:45:49,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:45:50,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000734367 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:45:58,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:45:58,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00428237 s, 956 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:46:00,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:46:00,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161248 s, 2.5 MB/s\n'; <rc> = 0
>Thread-13930::WARNING::2013-03-17 07:46:05,413::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:46:05,415::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:46:08,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:46:08,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761377 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:46:10,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:46:10,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898922 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:46:15,417::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:46:15,419::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:46:15,827::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:46:15,834::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:46:15,835::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:46:18,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:46:18,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184555 s, 222 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:46:20,227::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:46:20,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837674 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:46:28,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:46:28,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0092022 s, 445 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:46:30,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:46:30,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101206 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:46:38,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:46:38,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0046672 s, 878 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:46:40,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:46:40,399::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114145 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:46:49,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:46:49,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000664055 s, 6.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:46:50,439::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:46:50,481::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00456976 s, 896 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:46:59,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:46:59,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795142 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:47:00,522::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:47:00,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134672 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:47:09,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:47:09,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136996 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:47:10,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:47:10,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834503 s, 4.9 MB/s\n'; <rc> = 0
>Thread-13931::WARNING::2013-03-17 07:47:15,870::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:47:15,876::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:47:19,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:47:19,175::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000984983 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:47:20,684::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 07:47:20,686::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 07:47:20,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:47:20,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000869274 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:47:20,717::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000869274 s, 2.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 07:47:20,717::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 07:47:20,718::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 07:47:20,720::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 07:47:20,721::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:47:20,722::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:47:20,722::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:47:20,723::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:47:20,723::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:47:20,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:47:21,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:47:21,151::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:47:21,152::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 07:47:21,154::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 07:47:21,155::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 07:47:21,156::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 07:47:21,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:47:21,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:47:21,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:47:21,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149774 s, 273 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:47:25,879::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 07:47:25,880::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:47:25,881::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:47:26,298::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:47:26,305::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:47:26,306::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:47:29,182::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 07:47:29,185::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 07:47:29,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:47:29,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0126819 s, 161 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:47:29,236::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0126819 s, 161 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 07:47:29,237::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 07:47:29,237::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 07:47:29,238::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 07:47:29,239::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:47:29,240::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:47:29,240::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:47:29,241::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:47:29,241::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:47:29,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:47:29,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:47:29,682::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:47:29,686::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 07:47:29,688::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 07:47:29,689::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 07:47:29,690::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 07:47:29,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:47:30,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:47:30,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:47:30,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0079532 s, 515 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:47:31,614::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:47:31,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117281 s, 349 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:47:40,171::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:47:40,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000952805 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:47:41,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:47:41,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816775 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:47:50,209::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:47:50,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000706538 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:47:51,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:47:51,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144603 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:48:00,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:48:00,270::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00196649 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:48:01,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:48:01,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000931608 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:48:10,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:48:10,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136739 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:48:11,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:48:11,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151913 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:48:20,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:48:20,358::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143982 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:48:21,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:48:22,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878798 s, 4.7 MB/s\n'; <rc> = 0
>Thread-13932::WARNING::2013-03-17 07:48:26,345::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:48:26,348::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:48:30,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:48:30,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074654 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:48:32,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:48:32,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114783 s, 3.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:48:36,350::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:48:36,351::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:48:36,765::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:48:36,772::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:48:36,773::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:48:40,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:48:40,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130242 s, 314 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:48:42,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:48:42,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774643 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:48:50,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:48:50,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869985 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:48:52,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:48:52,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159465 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:49:00,481::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:49:00,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827646 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:49:02,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:49:02,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142625 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:49:10,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:49:10,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838986 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:49:12,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:49:12,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156989 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:49:20,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:49:20,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130137 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:49:22,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:49:22,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00679779 s, 603 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:49:30,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:49:30,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810029 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:49:32,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:49:32,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000999905 s, 4.1 MB/s\n'; <rc> = 0
>Thread-13933::WARNING::2013-03-17 07:49:36,802::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:49:36,806::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:49:40,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:49:40,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111082 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:49:42,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:49:42,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826631 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:49:46,812::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:49:46,813::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:49:47,284::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:49:47,296::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:49:47,297::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:49:50,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:49:50,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149576 s, 274 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:49:52,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:49:52,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840145 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:50:00,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:50:00,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862341 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:50:02,792::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:50:02,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000994749 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:50:10,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:50:10,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000978212 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:50:12,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:50:12,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803721 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:50:20,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:50:20,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850071 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:50:22,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:50:22,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000987856 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:50:30,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:50:30,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155247 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:50:32,997::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:50:33,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768803 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:50:40,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:50:40,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135672 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:50:43,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:50:43,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171219 s, 2.4 MB/s\n'; <rc> = 0
>Thread-13934::WARNING::2013-03-17 07:50:47,325::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:50:47,330::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:50:50,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:50:51,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911405 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:50:53,156::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:50:53,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129263 s, 3.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:50:57,334::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:50:57,335::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:50:57,748::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:50:57,755::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:50:57,757::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:51:01,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:51:01,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109147 s, 375 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:51:03,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:51:03,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145455 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:51:11,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:51:11,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827572 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:51:13,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:51:13,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000705617 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:51:21,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:51:21,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137719 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:51:23,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:51:23,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136879 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:51:31,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:51:31,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00743946 s, 551 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:51:33,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:51:33,494::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898407 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:51:41,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:51:41,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008597 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:51:43,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:51:43,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141231 s, 290 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:51:51,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:51:51,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149691 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:51:53,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:51:53,786::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000950788 s, 4.3 MB/s\n'; <rc> = 0
>Thread-13935::WARNING::2013-03-17 07:51:57,784::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:51:57,790::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:52:01,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:52:01,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834422 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:52:03,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:52:03,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845516 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:52:07,792::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:52:07,794::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:52:08,196::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:52:08,203::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:52:08,204::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:52:11,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:52:11,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102505 s, 400 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:52:13,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:52:13,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081862 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:52:21,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:52:21,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783304 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:52:23,973::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 07:52:23,975::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 07:52:23,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:52:24,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000888821 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:52:24,005::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000888821 s, 2.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 07:52:24,006::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 07:52:24,006::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 07:52:24,008::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 07:52:24,009::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:52:24,009::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:52:24,010::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:52:24,010::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:52:24,010::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:52:24,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:52:24,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:52:24,466::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:52:24,467::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 07:52:24,469::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 07:52:24,470::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 07:52:24,471::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 07:52:24,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:52:24,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:52:24,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:52:24,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150309 s, 273 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:52:31,384::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 07:52:31,386::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 07:52:31,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:52:31,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00759635 s, 270 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:52:31,421::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00759635 s, 270 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 07:52:31,422::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 07:52:31,422::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 07:52:31,424::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 07:52:31,425::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:52:31,425::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:52:31,426::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:52:31,426::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:52:31,426::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:52:31,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:52:31,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:52:31,805::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:52:31,806::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 07:52:31,808::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 07:52:31,809::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 07:52:31,810::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 07:52:31,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:52:32,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:52:32,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:52:32,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159262 s, 257 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:52:35,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:52:35,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000742497 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:52:42,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:52:42,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775196 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:52:45,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:52:45,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177005 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:52:52,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:52:52,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115276 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:52:55,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:52:55,238::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859408 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:53:02,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:53:02,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017478 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:53:05,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:53:05,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0097075 s, 422 kB/s\n'; <rc> = 0
>Thread-13936::WARNING::2013-03-17 07:53:08,233::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:53:08,241::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:53:12,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:53:12,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103567 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:53:15,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:53:15,395::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163513 s, 2.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:53:18,243::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 07:53:18,244::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:53:18,245::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:53:18,638::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:53:18,645::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:53:18,646::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:53:22,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:53:22,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122389 s, 335 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:53:25,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:53:25,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151502 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:53:32,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:53:32,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886624 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:53:35,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:53:35,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000724051 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:53:42,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:53:42,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823766 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:53:45,571::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:53:45,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762601 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:53:52,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:53:52,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00432389 s, 947 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:53:55,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:53:55,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162934 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:54:02,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:54:02,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00289013 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:54:05,732::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:54:05,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863719 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:54:12,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:54:12,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114056 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:54:15,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:54:15,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144592 s, 2.8 MB/s\n'; <rc> = 0
>Thread-13937::WARNING::2013-03-17 07:54:18,673::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:54:18,676::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:54:22,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:54:22,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135659 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:54:25,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:54:25,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124969 s, 328 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:54:28,678::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:54:28,680::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:54:29,089::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:54:29,123::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:54:29,125::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:54:32,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:54:32,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138911 s, 295 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:54:35,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:54:36,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117339 s, 349 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:54:42,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:54:42,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890472 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:54:46,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:54:46,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148993 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:54:52,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:54:52,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828969 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:54:56,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:54:56,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121537 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:55:02,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:55:02,820::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768624 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:55:06,209::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:55:06,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134428 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:55:12,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:55:12,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129807 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:55:16,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:55:16,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158723 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:55:22,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:55:22,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0030674 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:55:26,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:55:26,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116135 s, 3.5 MB/s\n'; <rc> = 0
>Thread-13938::WARNING::2013-03-17 07:55:29,150::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:55:29,153::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:55:32,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:55:32,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820065 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:55:36,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:55:36,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851757 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:55:39,158::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:55:39,159::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:55:39,568::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:55:39,575::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:55:39,576::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:55:42,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:55:43,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119484 s, 343 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:55:46,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:55:46,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810859 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:55:53,019::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:55:53,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763639 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:55:56,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:55:56,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819905 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:56:03,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:56:03,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895452 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:56:06,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:56:06,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000949514 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:56:13,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:56:13,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864053 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:56:16,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:56:16,777::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779194 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:56:23,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:56:23,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016265 s, 252 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:56:26,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:56:26,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144054 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:56:33,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:56:33,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155515 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:56:36,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:56:36,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851937 s, 4.8 MB/s\n'; <rc> = 0
>Thread-13939::WARNING::2013-03-17 07:56:39,601::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:56:39,603::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:56:43,221::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:56:43,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176926 s, 232 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:56:46,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:56:47,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128917 s, 318 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:56:49,607::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:56:49,608::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:56:50,016::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:56:50,023::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:56:50,024::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:56:53,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:56:53,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138555 s, 296 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:56:57,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:56:57,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136715 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:57:03,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:57:03,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184557 s, 222 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:57:07,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:57:07,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000730971 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:57:13,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:57:13,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149863 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:57:17,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:57:17,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000996113 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:57:23,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:57:23,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180372 s, 227 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:57:27,304::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 07:57:27,306::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 07:57:27,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:57:27,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000919055 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:57:27,338::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000919055 s, 2.2 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 07:57:27,338::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 07:57:27,338::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 07:57:27,340::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 07:57:27,341::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:57:27,342::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:57:27,342::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:57:27,342::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:57:27,343::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 07:57:27,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:57:27,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:57:27,825::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 07:57:27,826::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 07:57:27,829::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 07:57:27,830::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 07:57:27,830::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 07:57:27,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:57:28,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:57:28,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:57:28,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770422 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:57:33,489::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 07:57:33,491::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 07:57:33,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:57:33,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0167379 s, 122 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:57:33,548::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0167379 s, 122 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 07:57:33,549::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 07:57:33,549::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 07:57:33,551::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 07:57:33,552::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:57:33,552::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:57:33,553::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:57:33,553::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:57:33,554::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 07:57:33,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:57:33,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:57:33,984::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:57:33,985::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 07:57:33,988::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 07:57:33,989::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 07:57:33,989::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 07:57:33,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:57:34,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:57:34,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:57:34,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131025 s, 313 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:57:38,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:57:38,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153133 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:57:44,412::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:57:44,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0238704 s, 172 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:57:48,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:57:48,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00992184 s, 413 kB/s\n'; <rc> = 0
>Thread-13940::WARNING::2013-03-17 07:57:50,050::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:57:50,055::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:57:54,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:57:54,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728848 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:57:58,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:57:58,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886534 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:58:00,057::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:58:00,058::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:58:00,495::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:58:00,502::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:58:00,503::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:58:04,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:58:04,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0212441 s, 193 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:58:08,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:58:08,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872024 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:58:14,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:58:14,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850157 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:58:18,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:58:18,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154812 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:58:24,605::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:58:24,667::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193592 s, 212 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:58:28,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:58:28,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815838 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:58:34,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:58:34,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783135 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:58:38,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:58:38,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137381 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:58:44,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:58:44,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00496427 s, 825 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:58:48,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:58:48,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075555 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:58:54,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:58:54,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013463 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:58:58,922::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:58:58,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795275 s, 5.2 MB/s\n'; <rc> = 0
>Thread-13941::WARNING::2013-03-17 07:59:00,530::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 07:59:00,534::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 07:59:04,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:59:04,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00946368 s, 433 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:59:09,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:59:09,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912299 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 07:59:10,536::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 07:59:10,537::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 07:59:10,539::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 07:59:10,930::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 07:59:10,937::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 07:59:10,939::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 07:59:14,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:59:14,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133016 s, 308 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:59:19,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:59:19,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821582 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:59:24,894::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:59:24,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875694 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:59:29,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:59:29,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158239 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:59:34,938::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:59:34,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771184 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:59:39,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:59:39,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135333 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:59:44,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:59:45,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016424 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:59:49,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:59:49,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152927 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 07:59:55,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 07:59:55,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00569125 s, 720 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 07:59:59,612::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 07:59:59,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906217 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:00:05,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:00:05,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141649 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:00:09,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:00:09,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100445 s, 4.1 MB/s\n'; <rc> = 0
>Thread-13942::WARNING::2013-03-17 08:00:10,968::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:00:10,970::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 08:00:15,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:00:15,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00235496 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:00:19,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:00:19,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907491 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:00:20,973::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:00:20,974::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:00:21,468::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:00:21,474::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:00:21,476::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:00:25,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:00:25,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00900252 s, 455 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:00:30,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:00:30,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831308 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:00:35,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:00:35,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793226 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:00:40,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:00:40,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000967067 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:00:45,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:00:45,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841102 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:00:50,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:00:50,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755724 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:00:55,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:00:55,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792674 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:01:00,221::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:01:00,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083465 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:01:05,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:01:05,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001477 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:01:10,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:01:10,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00259236 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:01:15,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:01:15,398::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102761 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:01:20,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:01:20,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00595139 s, 688 kB/s\n'; <rc> = 0
>Thread-13943::WARNING::2013-03-17 08:01:21,507::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:01:21,511::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 08:01:25,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:01:25,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879373 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:01:30,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:01:30,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805429 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:01:31,513::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:01:31,515::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:01:31,908::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:01:31,915::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:01:31,917::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:01:35,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:01:35,481::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111678 s, 367 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:01:40,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:01:40,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860912 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:01:45,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:01:45,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884048 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:01:50,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:01:50,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836612 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:01:55,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:01:55,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728912 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:02:00,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:02:00,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00270299 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:02:05,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:02:05,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873575 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:02:10,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:02:10,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000666161 s, 6.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:02:15,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:02:15,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799201 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:02:20,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:02:20,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875232 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:02:25,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:02:25,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115504 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:02:31,043::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 08:02:31,045::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 08:02:31,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:02:31,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0167679 s, 122 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:02:31,222::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0167679 s, 122 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 08:02:31,226::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 08:02:31,227::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 08:02:31,232::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 08:02:31,239::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:02:31,243::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:02:31,245::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:02:31,247::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:02:31,250::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:02:31,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:02:31,765::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:02:31,785::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:02:31,790::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 08:02:31,803::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 08:02:31,806::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 08:02:31,808::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 08:02:31,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-13944::WARNING::2013-03-17 08:02:31,944::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:02:31,952::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:02:32,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:02:32,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:02:32,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00701287 s, 584 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:02:35,686::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 08:02:35,688::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 08:02:35,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:02:35,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0126546 s, 162 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:02:35,731::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0126546 s, 162 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 08:02:35,732::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 08:02:35,732::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 08:02:35,734::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 08:02:35,735::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:02:35,736::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:02:35,736::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:02:35,736::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:02:35,737::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:02:35,738::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:02:36,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:02:36,166::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:02:36,167::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 08:02:36,170::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 08:02:36,171::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 08:02:36,172::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 08:02:36,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:02:36,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:02:36,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:02:36,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189656 s, 216 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:02:41,955::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:02:41,957::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:02:42,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:02:42,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112631 s, 364 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:02:42,371::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:02:42,377::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:02:42,380::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:02:46,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:02:46,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124056 s, 330 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:02:52,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:02:52,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784188 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:02:56,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:02:56,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160606 s, 255 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:03:02,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:03:02,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00200993 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:03:06,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:03:06,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794768 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:03:12,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:03:12,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849246 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:03:16,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:03:16,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163776 s, 250 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:03:22,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:03:22,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000710837 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:03:26,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:03:26,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909731 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:03:32,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:03:32,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000889118 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:03:36,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:03:36,866::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109216 s, 375 kB/s\n'; <rc> = 0
>Thread-13945::WARNING::2013-03-17 08:03:42,437::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:03:42,442::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:03:42,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:03:42,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013786 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:03:46,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:03:46,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875915 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:03:52,445::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:03:52,446::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:03:52,862::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:03:52,869::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:03:52,870::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:03:52,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:03:52,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00855405 s, 479 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:03:56,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:03:56,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138678 s, 295 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:04:03,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:04:03,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815252 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:04:06,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:04:06,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793522 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:04:13,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:04:13,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000733857 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:04:16,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:04:17,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146945 s, 279 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:04:23,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:04:23,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149732 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:04:27,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:04:27,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805611 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:04:33,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:04:33,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00259677 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:04:37,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:04:37,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136601 s, 300 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:04:43,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:04:43,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149056 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:04:47,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:04:47,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137578 s, 3.0 MB/s\n'; <rc> = 0
>Thread-13946::WARNING::2013-03-17 08:04:52,900::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:04:52,902::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:04:53,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:04:53,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000969348 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:04:57,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:04:57,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116964 s, 350 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:05:02,908::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 08:05:02,909::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:05:02,910::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:05:03,353::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:05:03,359::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:05:03,361::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:05:03,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:05:03,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167324 s, 245 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:05:07,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:05:07,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149098 s, 275 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:05:13,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:05:13,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785426 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:05:17,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:05:17,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0200631 s, 204 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:05:23,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:05:23,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771967 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:05:27,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:05:27,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784141 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:05:33,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:05:33,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176549 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:05:37,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:05:37,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163744 s, 250 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:05:43,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:05:43,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837212 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:05:47,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:05:47,521::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152704 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:05:53,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:05:53,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135767 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:05:57,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:05:57,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013621 s, 301 kB/s\n'; <rc> = 0
>Thread-13947::WARNING::2013-03-17 08:06:03,388::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:06:03,391::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:06:03,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:06:03,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138098 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:06:07,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:06:07,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804603 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:06:13,394::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:06:13,395::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:06:13,793::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:06:13,800::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:06:13,802::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:06:14,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:06:14,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010459 s, 392 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:06:17,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:06:17,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156256 s, 262 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:06:24,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:06:24,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105823 s, 387 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:06:27,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:06:27,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141951 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:06:34,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:06:34,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841708 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:06:37,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:06:37,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135722 s, 302 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:06:44,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:06:44,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000946452 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:06:47,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:06:47,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167546 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:06:54,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:06:54,382::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151731 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:06:57,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:06:57,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00971585 s, 422 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:07:04,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:07:04,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147462 s, 278 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:07:07,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:07:07,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105026 s, 3.9 MB/s\n'; <rc> = 0
>Thread-13948::WARNING::2013-03-17 08:07:13,826::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:07:13,830::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:07:14,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:07:14,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101629 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:07:17,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:07:17,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00463001 s, 885 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:07:23,832::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:07:23,834::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:07:24,252::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:07:24,259::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:07:24,260::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:07:24,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:07:24,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00961376 s, 426 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:07:27,938::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:07:27,992::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107638 s, 381 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:07:34,667::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 08:07:34,668::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 08:07:34,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:07:34,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0012996 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:07:34,696::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0012996 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 08:07:34,697::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 08:07:34,697::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 08:07:34,699::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 08:07:34,700::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:07:34,700::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:07:34,701::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:07:34,703::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:07:34,703::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:07:34,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:07:35,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:07:35,124::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:07:35,125::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 08:07:35,127::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 08:07:35,129::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 08:07:35,129::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 08:07:35,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:07:35,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:07:35,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:07:35,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133529 s, 307 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:07:37,997::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 08:07:37,998::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 08:07:37,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:07:38,046::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0146964 s, 139 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:07:38,048::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0146964 s, 139 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 08:07:38,048::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 08:07:38,049::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 08:07:38,051::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 08:07:38,052::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:07:38,053::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:07:38,053::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:07:38,054::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:07:38,054::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:07:38,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:07:38,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:07:38,527::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:07:38,528::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 08:07:38,531::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 08:07:38,532::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 08:07:38,532::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 08:07:38,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:07:38,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:07:38,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:07:38,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138726 s, 295 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:07:45,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:07:45,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894656 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:07:48,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:07:48,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00233514 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:07:55,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:07:55,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080872 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:07:58,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:07:59,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129247 s, 317 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:08:05,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:08:05,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818084 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:08:09,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:08:09,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000938485 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:08:15,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:08:15,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837504 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:08:19,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:08:19,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145148 s, 2.8 MB/s\n'; <rc> = 0
>Thread-13949::WARNING::2013-03-17 08:08:24,290::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:08:24,294::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:08:25,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:08:25,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122393 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:08:29,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:08:29,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823772 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:08:34,296::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:08:34,298::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:08:34,707::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:08:34,714::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:08:34,715::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:08:35,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:08:36,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00661599 s, 619 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:08:39,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:08:39,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165457 s, 248 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:08:46,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:08:46,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815901 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:08:49,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:08:49,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137907 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:08:56,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:08:56,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786525 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:08:59,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:08:59,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106043 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:09:06,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:09:06,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896892 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:09:09,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:09:09,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000729711 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:09:16,280::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:09:16,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143218 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:09:19,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:09:19,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142072 s, 288 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:09:26,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:09:26,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000742164 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:09:29,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:09:29,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164531 s, 2.5 MB/s\n'; <rc> = 0
>Thread-13950::WARNING::2013-03-17 08:09:34,739::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:09:34,743::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:09:36,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:09:36,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836414 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:09:39,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:09:39,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144793 s, 2.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:09:44,745::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:09:44,747::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:09:45,158::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:09:45,165::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:09:45,166::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:09:46,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:09:46,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113298 s, 362 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:09:49,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:09:49,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126258 s, 324 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:09:56,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:09:56,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000969665 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:09:59,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:09:59,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145672 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:10:06,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:10:06,698::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913988 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:10:09,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:10:09,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822955 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:10:16,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:10:16,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145905 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:10:19,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:10:19,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936693 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:10:26,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:10:26,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857391 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:10:29,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:10:29,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143383 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:10:36,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:10:36,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845103 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:10:39,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:10:39,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163757 s, 2.5 MB/s\n'; <rc> = 0
>Thread-13951::WARNING::2013-03-17 08:10:45,193::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:10:45,197::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:10:46,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:10:46,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799286 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:10:49,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:10:49,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121356 s, 338 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:10:55,199::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 08:10:55,200::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:10:55,201::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:10:55,627::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:10:55,634::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:10:55,635::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:10:57,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:10:57,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00872095 s, 470 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:10:59,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:10:59,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00660991 s, 620 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:11:07,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:11:07,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00098883 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:11:09,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:11:09,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157356 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:11:17,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:11:17,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131065 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:11:19,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:11:19,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856787 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:11:27,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:11:27,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165811 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:11:29,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:11:29,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146053 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:11:37,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:11:37,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130152 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:11:39,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:11:40,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132396 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:11:47,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:11:47,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796851 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:11:50,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:11:50,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175422 s, 2.3 MB/s\n'; <rc> = 0
>Thread-13952::WARNING::2013-03-17 08:11:55,663::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:11:55,666::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:11:57,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:11:57,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822701 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:12:00,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:12:00,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148332 s, 2.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:12:05,668::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:12:05,669::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:12:06,180::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:12:06,189::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:12:06,191::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:12:07,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:12:07,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168027 s, 244 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:12:10,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:12:10,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013508 s, 303 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:12:17,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:12:17,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137727 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:12:20,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:12:20,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799553 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:12:27,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:12:27,765::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147125 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:12:30,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:12:30,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880219 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:12:37,803::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 08:12:37,805::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 08:12:37,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:12:37,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00125095 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:12:37,841::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00125095 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 08:12:37,841::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 08:12:37,841::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 08:12:37,843::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 08:12:37,844::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:12:37,845::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:12:37,845::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:12:37,845::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:12:37,846::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:12:37,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:12:38,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:12:38,287::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:12:38,288::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 08:12:38,292::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 08:12:38,293::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 08:12:38,294::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 08:12:38,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:12:38,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:12:38,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:12:38,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143158 s, 286 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:12:40,282::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 08:12:40,286::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 08:12:40,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:12:40,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0132552 s, 155 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:12:40,335::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0132552 s, 155 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 08:12:40,336::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 08:12:40,336::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 08:12:40,338::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 08:12:40,339::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:12:40,339::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:12:40,340::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:12:40,340::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:12:40,341::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:12:40,341::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:12:40,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:12:40,750::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:12:40,751::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 08:12:40,753::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 08:12:40,755::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 08:12:40,755::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 08:12:40,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:12:41,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:12:41,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:12:41,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160143 s, 256 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:12:48,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:12:48,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760905 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:12:51,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:12:51,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084742 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:12:58,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:12:58,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818012 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:13:01,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:13:01,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173685 s, 2.4 MB/s\n'; <rc> = 0
>Thread-13953::WARNING::2013-03-17 08:13:06,217::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:13:06,220::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:13:08,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:13:09,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102909 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:13:11,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:13:11,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917433 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:13:16,222::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:13:16,223::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:13:16,666::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:13:16,673::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:13:16,674::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:13:19,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:13:19,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0223128 s, 184 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:13:21,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:13:21,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0285009 s, 144 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:13:29,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:13:29,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819201 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:13:31,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:13:31,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162447 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:13:39,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:13:39,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752439 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:13:41,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:13:41,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000999442 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:13:49,280::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:13:49,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740011 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:13:51,429::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:13:51,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150395 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:13:59,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:13:59,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147601 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:14:01,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:14:01,521::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109025 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:14:09,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:14:09,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101586 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:14:11,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:14:11,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123171 s, 3.3 MB/s\n'; <rc> = 0
>Thread-13954::WARNING::2013-03-17 08:14:16,709::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:14:16,712::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:14:19,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:14:19,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155676 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:14:21,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:14:21,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149834 s, 273 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:14:26,720::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:14:26,722::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:14:27,288::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:14:27,297::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:14:27,299::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:14:29,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:14:29,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000701024 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:14:31,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:14:31,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156618 s, 262 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:14:39,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:14:39,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00302486 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:14:41,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:14:41,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150702 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:14:49,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:14:49,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807217 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:14:51,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:14:51,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186335 s, 220 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:14:59,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:14:59,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159144 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:15:01,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:15:01,809::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008533 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:15:09,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:15:09,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801843 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:15:11,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:15:11,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748164 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:15:19,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:15:20,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00263288 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:15:21,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:15:21,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000991416 s, 4.1 MB/s\n'; <rc> = 0
>Thread-13955::WARNING::2013-03-17 08:15:27,331::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:15:27,335::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:15:30,070::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:15:30,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810546 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:15:31,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:15:31,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766515 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:15:37,337::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:15:37,338::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:15:37,792::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:15:37,799::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:15:37,800::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:15:40,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:15:40,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887928 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:15:41,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:15:41,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147476 s, 278 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:15:50,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:15:50,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110584 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:15:51,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:15:52,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183679 s, 223 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:16:00,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:16:00,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178006 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:16:02,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:16:02,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130027 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:16:10,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:16:10,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079724 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:16:12,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:16:12,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00735277 s, 557 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:16:20,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:16:20,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152424 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:16:22,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:16:22,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014689 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:16:30,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:16:30,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017551 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:16:32,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:16:32,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175208 s, 2.3 MB/s\n'; <rc> = 0
>Thread-13956::WARNING::2013-03-17 08:16:37,826::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:16:37,830::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:16:40,596::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:16:40,622::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101445 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:16:42,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:16:42,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157001 s, 2.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:16:47,831::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 08:16:47,832::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:16:47,833::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:16:48,230::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:16:48,237::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:16:48,238::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:16:50,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:16:50,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145234 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:16:52,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:16:52,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014389 s, 285 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:17:00,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:17:00,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748595 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:17:02,341::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:17:02,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0030462 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:17:10,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:17:10,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159766 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:17:12,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:17:12,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014475 s, 283 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:17:20,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:17:20,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894202 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:17:22,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:17:22,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128099 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:17:30,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:17:31,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00206596 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:17:32,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:17:32,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101058 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:17:41,052::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 08:17:41,053::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 08:17:41,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:17:41,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00139372 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:17:41,095::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00139372 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 08:17:41,095::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 08:17:41,095::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 08:17:41,097::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 08:17:41,098::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:17:41,099::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:17:41,100::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:17:41,100::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:17:41,101::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:17:41,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:17:41,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:17:41,528::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:17:41,529::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 08:17:41,532::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 08:17:41,534::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 08:17:41,534::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 08:17:41,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:17:41,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:17:41,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:17:41,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00582575 s, 703 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:17:42,523::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 08:17:42,524::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 08:17:42,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:17:42,562::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00881823 s, 232 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:17:42,569::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00881823 s, 232 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 08:17:42,569::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 08:17:42,570::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 08:17:42,572::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 08:17:42,574::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:17:42,575::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:17:42,575::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:17:42,576::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:17:42,576::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:17:42,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:17:43,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:17:43,007::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:17:43,008::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 08:17:43,011::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 08:17:43,012::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 08:17:43,012::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 08:17:43,014::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:17:43,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:17:43,416::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:17:43,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179384 s, 228 kB/s\n'; <rc> = 0
>Thread-13957::WARNING::2013-03-17 08:17:48,271::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:17:48,276::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:17:51,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:17:52,017::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840339 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:17:53,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:17:53,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0195723 s, 209 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:17:58,278::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:17:58,279::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:17:58,699::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:17:58,706::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:17:58,707::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:18:02,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:18:02,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157124 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:18:03,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:18:03,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187746 s, 218 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:18:12,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:18:12,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000922196 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:18:13,592::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:18:13,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161976 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:18:22,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:18:22,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829952 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:18:23,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:18:23,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000998779 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:18:32,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:18:32,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774342 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:18:33,681::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:18:33,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878659 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:18:42,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:18:42,401::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00192568 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:18:43,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:18:43,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122797 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:18:52,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:18:52,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015203 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:18:53,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:18:53,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015796 s, 2.6 MB/s\n'; <rc> = 0
>Thread-13958::WARNING::2013-03-17 08:18:58,748::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:18:58,752::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:19:02,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:19:02,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119449 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:19:03,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:19:03,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759319 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:19:08,754::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:19:08,755::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:19:09,179::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:19:09,186::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:19:09,187::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:19:12,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:19:12,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117538 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:19:13,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:19:13,880::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142129 s, 288 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:19:22,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:19:22,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000965893 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:19:23,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:19:23,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132991 s, 308 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:19:32,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:19:32,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837742 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:19:33,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:19:33,979::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088199 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:19:42,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:19:42,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001487 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:19:43,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:19:44,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112841 s, 363 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:19:52,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:19:52,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104193 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:19:54,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:19:54,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130705 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:20:02,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:20:02,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137907 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:20:04,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:20:04,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150872 s, 271 kB/s\n'; <rc> = 0
>Thread-13959::WARNING::2013-03-17 08:20:09,216::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:20:09,221::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:20:13,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:20:13,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811554 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:20:14,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:20:14,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146694 s, 2.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:20:19,223::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:20:19,229::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:20:19,658::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:20:19,665::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:20:19,666::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:20:23,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:20:23,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919916 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:20:24,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:20:24,270::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203828 s, 201 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:20:33,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:20:33,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915547 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:20:34,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:20:34,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000722679 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:20:43,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:20:43,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102165 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:20:44,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:20:44,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172634 s, 237 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:20:53,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:20:53,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124167 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:20:54,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:20:54,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162057 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:21:03,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:21:03,424::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139251 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:21:04,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:21:04,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000996665 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:21:13,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:21:13,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127069 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:21:14,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:21:14,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100945 s, 4.1 MB/s\n'; <rc> = 0
>Thread-13960::WARNING::2013-03-17 08:21:19,698::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:21:19,701::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:21:23,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:21:23,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828246 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:21:24,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:21:24,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164025 s, 2.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:21:29,704::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:21:29,712::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:21:30,129::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:21:30,136::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:21:30,138::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:21:33,625::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:21:33,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017775 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:21:34,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:21:34,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111813 s, 366 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:21:43,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:21:43,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783211 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:21:44,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:21:44,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000997894 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:21:53,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:21:53,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170045 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:21:54,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:21:54,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818696 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:22:03,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:22:03,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743345 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:22:04,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:22:04,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0099018 s, 414 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:22:13,933::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:22:13,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138028 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:22:14,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:22:14,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152006 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:22:24,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:22:24,054::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00993445 s, 412 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:22:24,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:22:24,944::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00295966 s, 1.4 MB/s\n'; <rc> = 0
>Thread-13961::WARNING::2013-03-17 08:22:30,164::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:22:30,168::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:22:34,099::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:22:34,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879959 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:22:34,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:22:34,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812941 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:22:40,169::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 08:22:40,170::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:22:40,171::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:22:40,626::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:22:40,633::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:22:40,634::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:22:44,165::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 08:22:44,167::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 08:22:44,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:22:44,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0102948 s, 199 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:22:44,208::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0102948 s, 199 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 08:22:44,209::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 08:22:44,209::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 08:22:44,211::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 08:22:44,212::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:22:44,213::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:22:44,213::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:22:44,214::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:22:44,214::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:22:44,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:22:44,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:22:44,648::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:22:44,649::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 08:22:44,651::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 08:22:44,652::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 08:22:44,653::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 08:22:44,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:22:44,981::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 08:22:44,985::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 08:22:44,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:22:45,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:22:45,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:22:45,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0122196 s, 168 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:22:45,136::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0122196 s, 168 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 08:22:45,146::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 08:22:45,147::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 08:22:45,152::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 08:22:45,157::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:22:45,161::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:22:45,163::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00732058 s, 560 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:22:45,163::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:22:45,172::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:22:45,175::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:22:45,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:22:45,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:22:45,648::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:22:45,649::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 08:22:45,651::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 08:22:45,653::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 08:22:45,653::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 08:22:45,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:22:46,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:22:46,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:22:46,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125818 s, 326 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:22:55,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:22:55,311::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800655 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:22:56,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:22:56,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109957 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:23:05,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:23:05,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000715484 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:23:06,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:23:06,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161536 s, 254 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:23:15,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:23:15,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150454 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:23:16,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:23:16,188::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154854 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:23:25,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:23:25,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135177 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:23:26,194::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:23:26,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105872 s, 387 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:23:35,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:23:35,617::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800665 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:23:36,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:23:36,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140455 s, 2.9 MB/s\n'; <rc> = 0
>Thread-13962::WARNING::2013-03-17 08:23:40,664::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:23:40,668::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:23:45,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:23:45,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811033 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:23:46,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:23:46,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141467 s, 290 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:23:50,670::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:23:50,671::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:23:51,110::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:23:51,117::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:23:51,118::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:23:55,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:23:55,756::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074846 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:23:56,373::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:23:56,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152346 s, 269 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:24:05,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:24:05,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156312 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:24:06,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:24:06,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181127 s, 226 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:24:15,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:24:15,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807967 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:24:16,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:24:16,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759853 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:24:25,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:24:25,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143949 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:24:26,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:24:26,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017018 s, 241 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:24:36,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:24:36,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148163 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:24:36,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:24:36,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143594 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:24:46,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:24:46,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151151 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:24:46,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:24:46,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019392 s, 211 kB/s\n'; <rc> = 0
>Thread-13963::WARNING::2013-03-17 08:24:51,148::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:24:51,154::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:24:56,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:24:56,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00358043 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:24:56,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:24:56,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819026 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:25:01,157::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:25:01,158::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:25:01,681::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:25:01,687::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:25:01,688::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:25:06,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:25:06,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078474 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:25:06,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:25:06,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0570868 s, 71.8 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:25:16,373::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:25:16,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157343 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:25:16,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:25:16,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162759 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:25:26,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:25:26,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841218 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:25:26,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:25:26,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010245 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:25:36,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:25:36,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759037 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:25:36,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:25:36,950::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134077 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:25:46,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:25:46,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136215 s, 301 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:25:46,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:25:47,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00600315 s, 682 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:25:56,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:25:56,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146153 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:25:57,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:25:57,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917583 s, 4.5 MB/s\n'; <rc> = 0
>Thread-13964::WARNING::2013-03-17 08:26:01,724::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:26:01,727::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:26:06,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:26:06,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.049685 s, 82.4 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:26:07,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:26:07,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802915 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:26:11,730::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:26:11,731::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:26:12,207::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:26:12,213::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:26:12,215::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:26:16,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:26:16,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0099528 s, 412 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:26:17,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:26:17,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174148 s, 235 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:26:26,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:26:26,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826899 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:26:27,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:26:27,163::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000922698 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:26:37,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:26:37,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110926 s, 369 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:26:37,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:26:37,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150726 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:26:47,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:26:47,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155592 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:26:47,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:26:47,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866592 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:26:57,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:26:57,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084574 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:26:57,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:26:57,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823004 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:27:07,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:27:07,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:27:07,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0021081 s, 1.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:27:07,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110501 s, 3.7 MB/s\n'; <rc> = 0
>Thread-13965::WARNING::2013-03-17 08:27:12,247::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:27:12,256::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 08:27:17,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:27:17,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108344 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:27:17,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:27:17,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000930565 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:27:22,268::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:27:22,270::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:27:22,663::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:27:22,670::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:27:22,671::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:27:27,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:27:27,491::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192058 s, 213 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:27:27,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:27:27,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154227 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:27:37,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:27:37,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863522 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:27:37,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:27:37,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898943 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:27:47,529::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 08:27:47,531::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 08:27:47,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:27:47,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000906876 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:27:47,561::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000906876 s, 2.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 08:27:47,561::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 08:27:47,561::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 08:27:47,563::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 08:27:47,564::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:27:47,565::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:27:47,565::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:27:47,566::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:27:47,566::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:27:47,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:27:47,666::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 08:27:47,669::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 08:27:47,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:27:47,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000991111 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:27:47,720::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000991111 s, 2.1 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 08:27:47,721::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 08:27:47,721::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 08:27:47,723::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 08:27:47,725::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm reload operation' is holding the operation mutex, waiting...
>Thread-22::DEBUG::2013-03-17 08:27:48,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:27:48,017::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:27:48,024::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:27:48,024::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-20::DEBUG::2013-03-17 08:27:48,027::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:27:48,028::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:27:48,033::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:27:48,038::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::WARNING::2013-03-17 08:27:48,042::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 08:27:48,047::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 08:27:48,049::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 08:27:48,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:27:48,079::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:27:48,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:27:48,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:27:48,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:27:48,698::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:27:48,711::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 08:27:48,724::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 08:27:48,727::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 08:27:48,729::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 08:27:48,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:27:48,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167551 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:27:49,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:27:49,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:27:49,140::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112263 s, 365 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:27:58,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:27:58,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100349 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:27:59,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:27:59,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847827 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:28:08,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:28:08,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154434 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:28:09,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:28:09,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170816 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:28:18,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:28:18,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124794 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:28:19,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:28:19,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00070651 s, 5.8 MB/s\n'; <rc> = 0
>Thread-13966::WARNING::2013-03-17 08:28:22,709::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:28:22,715::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 08:28:28,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:28:28,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164697 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:28:29,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:28:29,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0019408 s, 2.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:28:32,717::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 08:28:32,718::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:28:32,719::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:28:33,198::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:28:33,205::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:28:33,206::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:28:38,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:28:39,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150507 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:28:39,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:28:39,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894523 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:28:49,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:28:49,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856982 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:28:49,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:28:49,593::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821029 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:28:59,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:28:59,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104642 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:28:59,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:28:59,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00239972 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:29:09,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:29:09,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872461 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:29:09,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:29:09,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868793 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:29:19,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:29:19,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088713 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:29:19,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:29:19,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140708 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:29:29,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:29:29,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163145 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:29:29,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:29:29,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766292 s, 5.3 MB/s\n'; <rc> = 0
>Thread-13967::WARNING::2013-03-17 08:29:33,235::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:29:33,240::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 08:29:39,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:29:39,236::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105956 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:29:39,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:29:39,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818627 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:29:43,242::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:29:43,244::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:29:43,665::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:29:43,671::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:29:43,672::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:29:49,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:29:49,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000973092 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:29:50,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:29:50,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150891 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:29:59,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:29:59,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109643 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:30:00,118::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:30:00,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141458 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:30:09,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:30:09,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796763 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:30:10,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:30:10,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147827 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:30:19,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:30:19,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151677 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:30:20,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:30:20,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167533 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:30:29,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:30:29,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179612 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:30:30,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:30:30,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013976 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:30:39,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:30:39,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154642 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:30:40,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:30:40,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149483 s, 2.7 MB/s\n'; <rc> = 0
>Thread-13968::WARNING::2013-03-17 08:30:43,697::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:30:43,703::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 08:30:49,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:30:49,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165642 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:30:50,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:30:50,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799963 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:30:53,707::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:30:53,709::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:30:54,119::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:30:54,125::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:30:54,127::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:30:59,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:30:59,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0007511 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:31:00,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:31:00,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150556 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:31:09,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:31:09,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775452 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:31:10,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:31:10,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000946022 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:31:19,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:31:19,611::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806737 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:31:20,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:31:20,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00090344 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:31:29,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:31:29,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124873 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:31:30,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:31:30,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825869 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:31:39,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:31:39,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916856 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:31:40,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:31:40,973::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873065 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:31:49,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:31:49,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138078 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:31:51,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:31:51,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779089 s, 5.3 MB/s\n'; <rc> = 0
>Thread-13969::WARNING::2013-03-17 08:31:54,171::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:31:54,176::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 08:31:59,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:31:59,786::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771751 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:32:01,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:32:01,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142428 s, 288 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:32:04,185::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:32:04,186::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:32:04,592::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:32:04,599::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:32:04,601::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:32:09,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:32:09,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121894 s, 336 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:32:11,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:32:11,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000903859 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:32:19,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:32:19,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162148 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:32:21,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:32:21,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830968 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:32:29,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:32:29,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789866 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:32:31,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:32:31,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830045 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:32:39,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:32:39,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139919 s, 293 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:32:41,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:32:41,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130618 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:32:49,978::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 08:32:49,984::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 08:32:49,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:32:50,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000876312 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:32:50,017::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000876312 s, 2.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 08:32:50,018::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 08:32:50,018::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 08:32:50,020::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 08:32:50,022::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:32:50,023::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:32:50,023::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:32:50,023::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:32:50,024::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:32:50,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:32:50,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:32:50,460::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:32:50,461::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 08:32:50,464::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 08:32:50,465::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 08:32:50,465::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 08:32:50,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:32:50,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:32:50,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:32:50,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190479 s, 215 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:32:51,460::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 08:32:51,462::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 08:32:51,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:32:51,512::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0117039 s, 175 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:32:51,514::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0117039 s, 175 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 08:32:51,514::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 08:32:51,514::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 08:32:51,522::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 08:32:51,527::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:32:51,528::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:32:51,528::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:32:51,529::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:32:51,529::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:32:51,530::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:32:51,973::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:32:51,981::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:32:51,982::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 08:32:51,984::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 08:32:51,986::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 08:32:51,986::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 08:32:51,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:32:52,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:32:52,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:32:52,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00410633 s, 997 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:33:00,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:33:00,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014104 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:33:02,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:33:02,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00250877 s, 1.6 MB/s\n'; <rc> = 0
>Thread-13970::WARNING::2013-03-17 08:33:04,632::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:33:04,636::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 08:33:10,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:33:10,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814774 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:33:12,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:33:12,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131224 s, 3.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:33:14,638::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:33:14,639::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:33:15,073::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:33:15,080::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:33:15,081::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:33:20,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:33:20,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791733 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:33:22,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:33:22,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832657 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:33:30,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:33:31,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00196995 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:33:32,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:33:32,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160884 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:33:41,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:33:41,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765173 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:33:42,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:33:42,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075776 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:33:51,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:33:51,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0035783 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:33:52,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:33:52,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015883 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:34:01,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:34:01,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016468 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:34:02,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:34:02,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170399 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:34:11,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:34:11,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136092 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:34:12,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:34:13,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083558 s, 4.9 MB/s\n'; <rc> = 0
>Thread-13971::WARNING::2013-03-17 08:34:15,125::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:34:15,128::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 08:34:21,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:34:21,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0037109 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:34:23,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:34:23,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102981 s, 4.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:34:25,130::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 08:34:25,131::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:34:25,133::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:34:25,558::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:34:25,565::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:34:25,566::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:34:31,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:34:31,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109843 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:34:33,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:34:33,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152439 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:34:41,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:34:41,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834143 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:34:43,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:34:43,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131627 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:34:51,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:34:51,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142902 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:34:53,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:34:53,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153312 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:35:01,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:35:01,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078846 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:35:03,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:35:03,413::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134997 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:35:11,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:35:11,448::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149829 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:35:13,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:35:13,491::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766366 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:35:21,455::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:35:21,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00897294 s, 456 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:35:23,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:35:23,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747025 s, 5.5 MB/s\n'; <rc> = 0
>Thread-13972::WARNING::2013-03-17 08:35:25,603::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:35:25,610::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 08:35:31,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:35:31,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788679 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:35:33,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:35:33,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837328 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:35:35,617::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:35:35,618::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:35:36,029::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:35:36,035::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:35:36,037::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:35:41,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:35:41,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850691 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:35:43,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:35:43,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133193 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:35:51,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:35:51,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165942 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:35:53,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:35:53,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148013 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:36:01,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:36:01,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783275 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:36:03,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:36:03,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133591 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:36:11,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:36:11,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00184055 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:36:13,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:36:14,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142529 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:36:21,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:36:21,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125696 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:36:24,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:36:24,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103796 s, 395 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:36:31,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:36:31,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143414 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:36:34,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:36:34,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111539 s, 3.7 MB/s\n'; <rc> = 0
>Thread-13973::WARNING::2013-03-17 08:36:36,064::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:36:36,068::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 08:36:41,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:36:41,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173864 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:36:44,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:36:44,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137628 s, 298 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:36:46,070::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:36:46,071::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:36:46,513::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:36:46,520::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:36:46,522::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:36:51,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:36:51,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00747891 s, 548 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:36:54,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:36:54,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849261 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:37:01,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:37:02,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00791091 s, 518 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:37:04,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:37:04,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188639 s, 217 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:37:12,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:37:12,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011674 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:37:14,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:37:14,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000720276 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:37:22,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:37:22,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103363 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:37:24,571::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:37:24,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848578 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:37:32,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:37:32,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100066 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:37:34,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:37:34,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759467 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:37:42,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:37:42,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013907 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:37:44,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:37:44,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00213513 s, 1.9 MB/s\n'; <rc> = 0
>Thread-13974::WARNING::2013-03-17 08:37:46,548::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:37:46,552::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 08:37:52,288::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 08:37:52,290::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 08:37:52,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:37:52,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000898193 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:37:52,319::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000898193 s, 2.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 08:37:52,320::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 08:37:52,320::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 08:37:52,322::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 08:37:52,323::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:37:52,324::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:37:52,324::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:37:52,324::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:37:52,325::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:37:52,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:37:52,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:37:52,766::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:37:52,767::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 08:37:52,769::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 08:37:52,770::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 08:37:52,771::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 08:37:52,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:37:53,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:37:53,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:37:53,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920051 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:37:54,810::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 08:37:54,812::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 08:37:54,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:37:54,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00986784 s, 208 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:37:54,864::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00986784 s, 208 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 08:37:54,864::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 08:37:54,865::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 08:37:54,866::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 08:37:54,867::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:37:54,868::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:37:54,869::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:37:54,869::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:37:54,869::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:37:54,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:37:55,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:37:55,311::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:37:55,312::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 08:37:55,315::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 08:37:55,316::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 08:37:55,317::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 08:37:55,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:37:55,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:37:55,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:37:55,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191512 s, 214 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:37:56,554::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:37:56,555::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:37:57,024::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:37:57,030::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:37:57,031::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:38:03,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:38:03,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017669 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:38:05,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:38:05,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0324424 s, 126 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:38:13,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:38:13,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129246 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:38:15,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:38:15,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145453 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:38:23,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:38:23,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150511 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:38:25,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:38:25,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078123 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:38:33,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:38:33,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156926 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:38:36,033::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:38:36,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158607 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:38:43,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:38:43,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790998 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:38:46,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:38:46,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000992131 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:38:53,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:38:53,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138478 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:38:56,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:38:56,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.023838 s, 172 kB/s\n'; <rc> = 0
>Thread-13975::WARNING::2013-03-17 08:38:57,062::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:38:57,067::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 08:39:03,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:39:03,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166919 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:39:06,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:39:06,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0256581 s, 160 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:39:07,071::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:39:07,072::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:39:07,504::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:39:07,510::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:39:07,512::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:39:13,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:39:13,529::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074156 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:39:16,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:39:16,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851489 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:39:23,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:39:23,569::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000996706 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:39:26,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:39:26,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0047082 s, 870 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:39:33,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:39:33,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891202 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:39:36,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:39:36,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790085 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:39:43,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:39:43,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796826 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:39:46,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:39:46,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167871 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:39:53,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:39:53,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102232 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:39:56,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:39:56,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148037 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:40:03,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:40:03,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141881 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:40:07,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:40:07,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106783 s, 384 kB/s\n'; <rc> = 0
>Thread-13976::WARNING::2013-03-17 08:40:07,555::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:40:07,563::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 08:40:13,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:40:13,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854453 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:40:17,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:40:17,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.030745 s, 133 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:40:17,566::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 08:40:17,568::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:40:17,569::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:40:18,006::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:40:18,012::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:40:18,014::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:40:23,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:40:23,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145747 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:40:27,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:40:27,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831386 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:40:33,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:40:33,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000947027 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:40:37,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:40:37,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907557 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:40:43,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:40:43,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00260639 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:40:47,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:40:47,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866297 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:40:53,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:40:53,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170479 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:40:57,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:40:57,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782656 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:41:03,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:41:03,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769452 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:41:07,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:41:07,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00202159 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:41:13,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:41:14,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000730131 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:41:17,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:41:17,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110907 s, 3.7 MB/s\n'; <rc> = 0
>Thread-13977::WARNING::2013-03-17 08:41:18,040::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:41:18,049::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 08:41:24,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:41:24,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122331 s, 335 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:41:28,066::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:41:28,067::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:41:28,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:41:28,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00714221 s, 573 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:41:28,507::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:41:28,513::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:41:28,514::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:41:34,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:41:34,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932605 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:41:38,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:41:38,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876505 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:41:44,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:41:44,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100242 s, 409 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:41:48,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:41:48,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000943752 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:41:54,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:41:54,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134191 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:41:58,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:41:58,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761394 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:42:04,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:42:04,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144732 s, 283 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:42:08,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:42:08,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187694 s, 218 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:42:14,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:42:14,358::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175846 s, 233 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:42:18,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:42:18,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174694 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:42:24,372::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:42:24,402::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000973243 s, 4.2 MB/s\n'; <rc> = 0
>Thread-13978::WARNING::2013-03-17 08:42:28,540::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:42:28,544::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:42:28,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:42:28,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131319 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:42:34,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:42:34,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130429 s, 314 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:42:38,546::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:42:38,547::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:42:38,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:42:38,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148841 s, 275 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:42:39,001::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:42:39,006::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:42:39,007::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:42:44,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:42:44,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189533 s, 216 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:42:48,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:42:48,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0025652 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:42:54,511::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 08:42:54,512::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 08:42:54,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:42:54,552::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0148813 s, 138 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:42:54,554::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0148813 s, 138 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 08:42:54,555::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 08:42:54,555::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 08:42:54,557::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 08:42:54,559::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:42:54,560::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:42:54,560::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:42:54,561::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:42:54,561::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:42:54,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:42:54,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:42:54,990::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:42:54,991::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 08:42:54,993::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 08:42:54,994::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 08:42:54,995::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 08:42:54,997::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:42:55,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:42:55,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:42:55,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112359 s, 365 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:42:58,907::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 08:42:58,909::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 08:42:58,910::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:42:58,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0102264 s, 200 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:42:58,953::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0102264 s, 200 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 08:42:58,953::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 08:42:58,954::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 08:42:58,955::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 08:42:58,956::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:42:58,957::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:42:58,958::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:42:58,958::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:42:58,959::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:42:58,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:42:59,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:42:59,393::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:42:59,394::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 08:42:59,396::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 08:42:59,397::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 08:42:59,398::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 08:42:59,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:42:59,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:42:59,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:42:59,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00966687 s, 424 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:43:05,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:43:05,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100667 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:43:09,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:43:09,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000988227 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:43:15,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:43:15,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841025 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:43:19,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:43:19,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141031 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:43:25,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:43:25,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162009 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:43:29,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:43:30,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00073631 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:43:35,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:43:35,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013569 s, 3.0 MB/s\n'; <rc> = 0
>Thread-13979::WARNING::2013-03-17 08:43:39,034::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:43:39,038::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:43:40,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:43:40,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089527 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:43:45,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:43:45,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793436 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:43:49,040::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:43:49,041::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:43:49,453::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:43:49,460::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:43:49,461::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:43:50,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:43:50,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00950779 s, 431 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:43:55,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:43:55,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163121 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:44:00,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:44:00,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924399 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:44:05,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:44:05,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152772 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:44:10,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:44:10,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155471 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:44:15,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:44:15,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077064 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:44:20,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:44:20,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809513 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:44:25,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:44:25,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850011 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:44:30,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:44:30,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861441 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:44:35,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:44:35,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00347966 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:44:40,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:44:40,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825148 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:44:45,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:44:45,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000961522 s, 4.3 MB/s\n'; <rc> = 0
>Thread-13980::WARNING::2013-03-17 08:44:49,507::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:44:49,510::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:44:50,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:44:50,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831598 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:44:55,944::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:44:55,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085256 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:44:59,512::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:44:59,514::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:44:59,989::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:44:59,996::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:44:59,997::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:45:00,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:45:00,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137875 s, 297 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:45:05,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:45:06,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777949 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:45:10,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:45:10,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190535 s, 215 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:45:16,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:45:16,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125834 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:45:20,854::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:45:20,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125151 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:45:26,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:45:26,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176892 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:45:30,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:45:30,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130764 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:45:36,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:45:36,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787043 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:45:41,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:45:41,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809771 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:45:46,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:45:46,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00516569 s, 793 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:45:51,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:45:51,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000910335 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:45:56,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:45:56,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167379 s, 2.4 MB/s\n'; <rc> = 0
>Thread-13981::WARNING::2013-03-17 08:46:00,023::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:46:00,025::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:46:01,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:46:01,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000719901 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:46:06,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:46:06,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904559 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:46:10,027::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 08:46:10,028::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:46:10,029::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:46:10,467::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:46:10,473::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:46:10,475::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:46:11,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:46:11,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107772 s, 380 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:46:16,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:46:16,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808057 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:46:21,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:46:21,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000956477 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:46:26,292::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:46:26,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141891 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:46:31,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:46:31,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168443 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:46:36,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:46:36,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813165 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:46:41,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:46:41,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759316 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:46:46,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:46:46,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168044 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:46:51,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:46:51,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154473 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:46:56,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:46:56,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014291 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:47:01,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:47:01,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176586 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:47:06,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:47:06,622::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00887298 s, 462 kB/s\n'; <rc> = 0
>Thread-13982::WARNING::2013-03-17 08:47:10,505::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:47:10,507::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:47:11,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:47:11,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172137 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:47:16,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:47:16,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855297 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:47:20,513::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:47:20,514::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:47:20,914::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:47:20,920::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:47:20,922::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:47:21,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:47:21,793::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125919 s, 325 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:47:26,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:47:26,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116019 s, 353 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:47:31,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:47:31,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841093 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:47:36,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:47:36,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920421 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:47:41,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:47:41,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154258 s, 266 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:47:46,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:47:46,786::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809305 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:47:51,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:47:52,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149815 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:47:56,793::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 08:47:56,794::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 08:47:56,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:47:56,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00121111 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:47:56,827::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00121111 s, 1.7 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 08:47:56,828::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 08:47:56,828::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 08:47:56,830::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 08:47:56,832::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:47:56,832::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:47:56,833::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:47:56,833::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:47:56,834::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:47:56,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:47:57,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:47:57,288::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:47:57,289::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 08:47:57,292::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 08:47:57,293::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 08:47:57,293::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 08:47:57,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:47:57,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:47:57,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:47:57,793::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129232 s, 317 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:48:02,068::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 08:48:02,069::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 08:48:02,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:48:02,130::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0175102 s, 117 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:48:02,132::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0175102 s, 117 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 08:48:02,135::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 08:48:02,135::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 08:48:02,137::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 08:48:02,138::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:48:02,139::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:48:02,139::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:48:02,139::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:48:02,140::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:48:02,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:48:02,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:48:02,596::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:48:02,597::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 08:48:02,599::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 08:48:02,601::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 08:48:02,601::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 08:48:02,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:48:02,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:48:02,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:48:03,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00743099 s, 551 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:48:07,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:48:07,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0282677 s, 145 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:48:13,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:48:13,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881217 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:48:17,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:48:17,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774815 s, 5.3 MB/s\n'; <rc> = 0
>Thread-13983::WARNING::2013-03-17 08:48:20,959::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:48:20,962::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:48:23,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:48:23,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000729647 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:48:27,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:48:27,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874695 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:48:30,964::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:48:30,965::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:48:31,385::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:48:31,392::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:48:31,393::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:48:33,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:48:33,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149764 s, 273 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:48:37,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:48:37,973::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141674 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:48:43,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:48:43,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842223 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:48:47,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:48:48,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00072193 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:48:53,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:48:53,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836222 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:48:58,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:48:58,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799209 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:49:03,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:49:03,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154574 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:49:08,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:49:08,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786627 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:49:13,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:49:13,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086873 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:49:18,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:49:18,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00191672 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:49:23,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:49:23,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138425 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:49:28,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:49:28,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0225419 s, 182 kB/s\n'; <rc> = 0
>Thread-13984::WARNING::2013-03-17 08:49:31,427::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:49:31,431::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:49:33,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:49:33,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763954 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:49:38,345::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:49:38,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818553 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:49:41,433::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:49:41,434::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:49:41,861::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:49:41,868::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:49:41,869::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:49:43,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:49:43,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100228 s, 409 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:49:48,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:49:48,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778557 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:49:53,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:49:53,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0281004 s, 146 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:49:58,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:49:58,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000967224 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:50:03,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:50:03,928::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000693081 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:50:08,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:50:08,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129427 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:50:13,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:50:14,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163663 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:50:18,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:50:18,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172092 s, 238 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:50:24,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:50:24,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00938442 s, 436 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:50:28,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:50:28,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146264 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:50:34,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:50:34,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0528999 s, 77.4 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:50:38,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:50:38,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000990042 s, 4.1 MB/s\n'; <rc> = 0
>Thread-13985::WARNING::2013-03-17 08:50:41,897::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:50:41,903::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:50:44,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:50:44,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00829487 s, 494 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:50:48,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:50:48,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809512 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:50:51,907::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:50:51,908::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:50:52,364::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:50:52,371::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:50:52,373::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:50:54,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:50:54,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139123 s, 294 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:50:58,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:50:58,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110715 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:51:04,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:51:04,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131027 s, 313 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:51:08,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:51:08,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109529 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:51:14,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:51:14,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827534 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:51:18,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:51:18,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000989195 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:51:24,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:51:24,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162688 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:51:28,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:51:28,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117242 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:51:34,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:51:34,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142671 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:51:38,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:51:38,979::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00825732 s, 496 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:51:44,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:51:44,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141795 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:51:48,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:51:49,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000727711 s, 5.6 MB/s\n'; <rc> = 0
>Thread-13986::WARNING::2013-03-17 08:51:52,399::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:51:52,403::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:51:54,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:51:54,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872207 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:51:59,033::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:51:59,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000718275 s, 5.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:52:02,405::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 08:52:02,406::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:52:02,407::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:52:02,821::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:52:02,828::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:52:02,829::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:52:04,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:52:04,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0198974 s, 206 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:52:09,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:52:09,238::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0257538 s, 159 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:52:15,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:52:15,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815942 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:52:19,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:52:19,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144057 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:52:25,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:52:25,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858852 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:52:29,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:52:29,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00874334 s, 468 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:52:35,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:52:35,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880789 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:52:39,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:52:39,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136081 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:52:45,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:52:45,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135213 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:52:49,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:52:49,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00954731 s, 429 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:52:55,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:52:55,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149616 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:52:59,486::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 08:52:59,487::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 08:52:59,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:52:59,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00135008 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:52:59,526::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00135008 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 08:52:59,527::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 08:52:59,527::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 08:52:59,529::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 08:52:59,534::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:52:59,535::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:52:59,535::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:52:59,535::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:52:59,536::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:52:59,537::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:52:59,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:52:59,973::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:52:59,974::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 08:52:59,977::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 08:52:59,978::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 08:52:59,979::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 08:52:59,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:53:00,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:53:00,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:53:00,401::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00501897 s, 816 kB/s\n'; <rc> = 0
>Thread-13987::WARNING::2013-03-17 08:53:02,859::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:53:02,867::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:53:05,396::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 08:53:05,397::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 08:53:05,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:53:05,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00151104 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:53:05,432::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00151104 s, 1.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 08:53:05,432::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 08:53:05,433::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 08:53:05,434::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 08:53:05,435::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:53:05,436::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:53:05,436::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:53:05,437::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:53:05,437::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:53:05,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:53:05,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:53:05,841::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:53:05,842::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 08:53:05,844::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 08:53:05,846::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 08:53:05,846::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 08:53:05,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:53:06,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:53:06,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:53:06,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00998671 s, 410 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:53:10,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:53:10,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133647 s, 306 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:53:12,870::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:53:12,873::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:53:13,290::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:53:13,297::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:53:13,298::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:53:16,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:53:16,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189939 s, 216 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:53:20,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:53:20,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757838 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:53:26,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:53:26,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155356 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:53:30,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:53:30,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00251963 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:53:36,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:53:36,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858357 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:53:40,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:53:40,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864759 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:53:46,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:53:46,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000953968 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:53:50,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:53:50,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141395 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:53:56,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:53:56,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163439 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:54:00,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:54:00,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143597 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:54:06,681::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:54:06,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159769 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:54:10,697::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:54:10,739::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142972 s, 2.9 MB/s\n'; <rc> = 0
>Thread-13988::WARNING::2013-03-17 08:54:13,347::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:54:13,350::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:54:16,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:54:16,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000729103 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:54:20,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:54:20,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822435 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:54:23,352::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:54:23,353::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:54:23,788::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:54:23,795::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:54:23,796::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:54:26,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:54:26,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147162 s, 278 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:54:30,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:54:30,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000697558 s, 5.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:54:36,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:54:36,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896768 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:54:40,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:54:40,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141001 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:54:46,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:54:47,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816566 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:54:50,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:54:50,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087719 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:54:57,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:54:57,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871928 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:55:00,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:55:00,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168524 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:55:07,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:55:07,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177611 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:55:10,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:55:10,992::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140232 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:55:17,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:55:17,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872619 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:55:20,997::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:55:21,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173321 s, 2.4 MB/s\n'; <rc> = 0
>Thread-13989::WARNING::2013-03-17 08:55:23,823::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:55:23,827::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:55:27,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:55:27,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920591 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:55:31,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:55:31,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881782 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:55:33,833::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:55:33,834::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:55:34,291::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:55:34,298::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:55:34,300::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:55:37,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:55:37,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149216 s, 275 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:55:41,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:55:41,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148729 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:55:47,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:55:47,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132266 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:55:51,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:55:51,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167975 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:55:57,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:55:57,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105068 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:56:01,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:56:01,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146686 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:56:07,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:56:07,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151187 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:56:11,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:56:11,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820522 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:56:17,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:56:17,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010171 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:56:21,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:56:21,309::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152185 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:56:27,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:56:27,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00322818 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:56:31,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:56:31,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875468 s, 4.7 MB/s\n'; <rc> = 0
>Thread-13990::WARNING::2013-03-17 08:56:34,329::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:56:34,338::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:56:37,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:56:37,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173685 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:56:41,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:56:41,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000974776 s, 4.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:56:44,351::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:56:44,353::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:56:44,763::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:56:44,772::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:56:44,774::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:56:47,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:56:48,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100497 s, 408 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:56:51,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:56:51,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749127 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:56:58,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:56:58,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829626 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:57:01,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:57:01,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809287 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:57:08,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:57:08,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775635 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:57:11,471::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:57:11,512::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118924 s, 344 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:57:18,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:57:18,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148356 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:57:21,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:57:21,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188222 s, 218 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:57:28,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:57:28,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777448 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:57:31,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:57:31,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809137 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:57:38,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:57:38,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162792 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:57:41,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:57:41,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000977949 s, 4.2 MB/s\n'; <rc> = 0
>Thread-13991::WARNING::2013-03-17 08:57:44,803::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:57:44,806::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:57:48,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:57:48,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870014 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:57:51,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:57:51,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913703 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:57:54,817::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 08:57:54,818::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:57:54,819::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:57:55,264::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:57:55,270::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:57:55,272::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:57:58,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:57:58,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119381 s, 343 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:58:01,693::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 08:58:01,695::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 08:58:01,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:58:01,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0111973 s, 183 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:58:01,754::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0111973 s, 183 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 08:58:01,754::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 08:58:01,755::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 08:58:01,757::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 08:58:01,758::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:58:01,758::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:58:01,759::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:58:01,759::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:58:01,760::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 08:58:01,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:58:02,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:58:02,231::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 08:58:02,232::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 08:58:02,236::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 08:58:02,237::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 08:58:02,238::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 08:58:02,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:58:02,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:58:02,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:58:02,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178836 s, 229 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:58:08,590::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 08:58:08,591::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 08:58:08,592::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:58:08,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00080476 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:58:08,618::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00080476 s, 2.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 08:58:08,618::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 08:58:08,618::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 08:58:08,620::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 08:58:08,621::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:58:08,622::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:58:08,622::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:58:08,623::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:58:08,623::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 08:58:08,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:58:09,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:58:09,066::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:58:09,067::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 08:58:09,070::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 08:58:09,071::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 08:58:09,072::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 08:58:09,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:58:09,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:58:09,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:58:09,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130133 s, 315 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:58:12,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:58:12,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010046 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:58:19,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:58:19,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820417 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:58:22,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:58:22,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744175 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:58:29,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:58:29,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107346 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:58:32,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:58:32,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162327 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:58:39,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:58:39,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146171 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:58:42,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:58:42,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804457 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:58:49,781::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:58:49,820::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145123 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:58:52,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:58:52,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101102 s, 4.1 MB/s\n'; <rc> = 0
>Thread-13992::WARNING::2013-03-17 08:58:55,298::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 08:58:55,302::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 08:58:59,861::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:58:59,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826814 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:59:02,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:59:02,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876481 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 08:59:05,306::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 08:59:05,307::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 08:59:05,693::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 08:59:05,700::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 08:59:05,701::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 08:59:09,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:59:09,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204143 s, 201 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:59:12,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:59:12,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085888 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:59:20,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:59:20,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795599 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:59:23,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:59:23,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759612 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:59:30,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:59:30,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137155 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:59:33,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:59:33,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855921 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:59:40,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:59:40,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159072 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:59:43,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:59:43,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828914 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 08:59:50,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 08:59:50,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168411 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 08:59:53,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 08:59:53,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155139 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:00:00,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:00:00,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814654 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:00:03,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:00:03,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116946 s, 3.5 MB/s\n'; <rc> = 0
>Thread-13993::WARNING::2013-03-17 09:00:05,727::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:00:05,731::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:00:10,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:00:10,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114739 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:00:13,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:00:13,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145457 s, 2.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:00:15,772::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:00:15,774::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:00:16,268::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:00:16,274::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:00:16,276::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:00:20,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:00:20,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127952 s, 320 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:00:23,260::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:00:23,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000731526 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:00:30,592::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:00:30,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163033 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:00:33,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:00:33,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152285 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:00:40,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:00:40,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000889714 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:00:43,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:00:43,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150593 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:00:50,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:00:50,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176707 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:00:53,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:00:53,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074685 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:01:00,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:01:00,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161634 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:01:03,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:01:03,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00291879 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:01:10,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:01:10,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821725 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:01:13,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:01:13,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107842 s, 3.8 MB/s\n'; <rc> = 0
>Thread-13994::WARNING::2013-03-17 09:01:16,306::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:01:16,316::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:01:21,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:01:21,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151957 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:01:23,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:01:23,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000599416 s, 6.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:01:26,333::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:01:26,335::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:01:26,816::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:01:26,823::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:01:26,824::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:01:31,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:01:31,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137985 s, 297 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:01:33,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:01:33,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000933496 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:01:41,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:01:41,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824141 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:01:43,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:01:43,607::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788048 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:01:51,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:01:51,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138865 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:01:53,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:01:53,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173722 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:02:01,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:02:01,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157091 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:02:03,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:02:03,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137979 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:02:11,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:02:11,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159446 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:02:13,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:02:13,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.007993 s, 512 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:02:21,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:02:21,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000966725 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:02:23,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:02:23,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0078611 s, 521 kB/s\n'; <rc> = 0
>Thread-13995::WARNING::2013-03-17 09:02:26,850::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:02:26,854::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:02:31,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:02:31,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00218148 s, 1.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:02:33,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:02:33,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162905 s, 2.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:02:36,856::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:02:36,858::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:02:37,279::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:02:37,286::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:02:37,287::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:02:41,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:02:41,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00874414 s, 468 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:02:43,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:02:43,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181626 s, 226 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:02:51,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:02:51,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142218 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:02:53,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:02:54,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00795542 s, 515 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:03:01,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:03:01,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128478 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:03:04,027::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 09:03:04,028::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 09:03:04,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:03:04,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0108884 s, 188 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:03:04,092::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0108884 s, 188 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 09:03:04,092::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 09:03:04,092::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 09:03:04,095::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 09:03:04,096::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:03:04,097::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:03:04,097::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:03:04,098::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:03:04,098::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:03:04,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:03:04,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:03:04,509::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:03:04,510::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 09:03:04,512::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 09:03:04,513::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 09:03:04,514::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 09:03:04,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:03:04,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:03:04,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:03:04,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100942 s, 406 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:03:11,890::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 09:03:11,891::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 09:03:11,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:03:11,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000763408 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:03:11,924::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000763408 s, 2.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 09:03:11,924::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 09:03:11,925::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 09:03:11,926::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 09:03:11,927::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:03:11,928::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:03:11,928::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:03:11,929::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:03:11,929::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:03:11,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:03:12,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:03:12,357::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:03:12,358::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 09:03:12,360::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 09:03:12,361::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 09:03:12,362::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 09:03:12,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:03:12,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:03:12,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:03:12,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108453 s, 378 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:03:14,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:03:15,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838806 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:03:22,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:03:22,907::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000939355 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:03:25,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:03:25,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014903 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:03:32,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:03:33,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158231 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:03:35,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:03:35,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114303 s, 3.6 MB/s\n'; <rc> = 0
>Thread-13996::WARNING::2013-03-17 09:03:37,315::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:03:37,321::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:03:43,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:03:43,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807966 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:03:45,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:03:45,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000984385 s, 4.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:03:47,324::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 09:03:47,326::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:03:47,327::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:03:47,810::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:03:47,817::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:03:47,818::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:03:53,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:03:53,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126371 s, 324 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:03:55,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:03:55,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860435 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:04:03,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:04:03,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835633 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:04:05,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:04:05,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136087 s, 301 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:04:13,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:04:13,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00228304 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:04:15,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:04:15,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0293674 s, 139 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:04:23,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:04:23,424::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167746 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:04:25,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:04:25,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804979 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:04:33,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:04:33,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166679 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:04:35,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:04:35,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013598 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:04:43,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:04:43,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104804 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:04:45,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:04:45,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164542 s, 2.5 MB/s\n'; <rc> = 0
>Thread-13997::WARNING::2013-03-17 09:04:47,842::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:04:47,846::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:04:53,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:04:53,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148826 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:04:55,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:04:55,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00182612 s, 2.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:04:57,849::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:04:57,850::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:04:58,256::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:04:58,263::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:04:58,264::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:05:03,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:05:03,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752221 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:05:05,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:05:05,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864714 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:05:13,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:05:13,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785637 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:05:15,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:05:15,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862235 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:05:23,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:05:23,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00307622 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:05:25,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:05:25,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00284638 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:05:33,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:05:33,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769039 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:05:35,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:05:35,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102376 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:05:44,021::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:05:44,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00953501 s, 430 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:05:45,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:05:45,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0033431 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:05:54,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:05:54,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139169 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:05:55,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:05:55,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741485 s, 5.5 MB/s\n'; <rc> = 0
>Thread-13998::WARNING::2013-03-17 09:05:58,300::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:05:58,306::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:06:04,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:06:04,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00829651 s, 494 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:06:05,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:06:05,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103015 s, 4.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:06:08,309::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:06:08,310::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:06:08,714::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:06:08,721::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:06:08,722::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:06:14,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:06:14,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178495 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:06:15,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:06:15,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154693 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:06:24,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:06:24,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010761 s, 381 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:06:25,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:06:25,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146711 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:06:34,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:06:34,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113136 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:06:35,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:06:35,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176981 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:06:44,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:06:44,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122423 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:06:45,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:06:45,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770861 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:06:54,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:06:54,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160612 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:06:55,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:06:56,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140638 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:07:04,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:07:04,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163481 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:07:06,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:07:06,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859052 s, 4.8 MB/s\n'; <rc> = 0
>Thread-13999::WARNING::2013-03-17 09:07:08,748::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:07:08,751::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:07:14,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:07:14,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085298 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:07:16,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:07:16,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155932 s, 2.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:07:18,754::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:07:18,755::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:07:19,148::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:07:19,155::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:07:19,156::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:07:24,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:07:24,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788039 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:07:26,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:07:26,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144657 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:07:34,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:07:34,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148847 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:07:36,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:07:36,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809192 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:07:44,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:07:44,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140167 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:07:46,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:07:46,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000998696 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:07:55,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:07:55,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129634 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:07:56,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:07:56,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154915 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:08:05,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:08:05,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134682 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:08:06,283::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 09:08:06,285::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 09:08:06,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:08:06,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00140508 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:08:06,326::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00140508 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 09:08:06,327::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 09:08:06,327::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 09:08:06,329::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 09:08:06,330::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:08:06,330::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:08:06,331::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:08:06,331::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:08:06,331::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:08:06,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:08:06,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:08:06,776::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:08:06,777::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 09:08:06,780::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 09:08:06,781::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 09:08:06,781::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 09:08:06,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:08:07,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:08:07,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:08:07,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00709426 s, 577 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:08:15,193::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 09:08:15,195::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 09:08:15,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:08:15,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0157912 s, 130 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:08:15,256::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0157912 s, 130 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 09:08:15,256::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 09:08:15,256::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 09:08:15,258::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 09:08:15,259::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:08:15,260::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:08:15,260::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:08:15,260::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:08:15,261::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:08:15,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:08:15,700::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:08:15,708::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:08:15,710::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 09:08:15,716::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 09:08:15,717::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 09:08:15,717::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 09:08:15,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:08:16,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:08:16,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:08:16,132::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105071 s, 390 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:08:17,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:08:17,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813651 s, 5.0 MB/s\n'; <rc> = 0
>Thread-14000::WARNING::2013-03-17 09:08:19,186::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:08:19,192::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:08:26,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:08:26,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821903 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:08:27,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:08:27,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798648 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:08:29,195::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:08:29,196::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:08:29,628::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:08:29,635::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:08:29,636::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:08:36,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:08:36,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086404 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:08:37,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:08:37,404::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144999 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:08:46,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:08:46,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807678 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:08:47,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:08:47,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839379 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:08:56,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:08:56,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154886 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:08:57,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:08:57,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858504 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:09:06,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:09:06,490::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015674 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:09:07,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:09:07,521::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00317898 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:09:16,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:09:16,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131265 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:09:17,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:09:17,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769913 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:09:26,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:09:26,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141602 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:09:27,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:09:27,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143113 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14001::WARNING::2013-03-17 09:09:29,668::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:09:29,674::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:09:36,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:09:36,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788004 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:09:37,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:09:37,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104107 s, 3.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:09:39,678::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 09:09:39,680::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:09:39,681::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:09:40,071::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:09:40,077::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:09:40,079::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:09:46,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:09:46,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000722067 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:09:47,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:09:47,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150683 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:09:56,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:09:56,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000731828 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:09:57,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:09:57,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919638 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:10:07,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:10:07,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00569788 s, 719 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:10:07,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:10:07,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000727304 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:10:17,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:10:17,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748682 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:10:17,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:10:17,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144375 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:10:27,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:10:27,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149728 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:10:27,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:10:27,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150439 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:10:37,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:10:37,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000697807 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:10:37,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:10:37,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117317 s, 3.5 MB/s\n'; <rc> = 0
>Thread-14002::WARNING::2013-03-17 09:10:40,106::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:10:40,110::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:10:47,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:10:47,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806077 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:10:47,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:10:47,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817582 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:10:50,112::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:10:50,113::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:10:50,564::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:10:50,570::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:10:50,572::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:10:57,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:10:57,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014166 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:10:57,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:10:57,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795618 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:11:07,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:11:07,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123693 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:11:07,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:11:08,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902265 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:11:17,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:11:17,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115084 s, 356 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:11:18,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:11:18,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076102 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:11:27,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:11:27,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155753 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:11:28,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:11:28,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000979171 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:11:37,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:11:37,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904857 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:11:38,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:11:38,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144133 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:11:47,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:11:47,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0036645 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:11:48,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:11:48,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000676268 s, 6.1 MB/s\n'; <rc> = 0
>Thread-14003::WARNING::2013-03-17 09:11:50,602::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:11:50,608::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:11:57,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:11:57,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010028 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:11:58,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:11:58,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828884 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:12:00,611::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:12:00,612::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:12:01,037::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:12:01,044::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:12:01,046::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:12:07,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:12:07,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000990317 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:12:08,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:12:08,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792711 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:12:18,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:12:18,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0214216 s, 191 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:12:18,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:12:18,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111527 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:12:28,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:12:28,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088898 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:12:28,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:12:28,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860263 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:12:38,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:12:38,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853479 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:12:38,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:12:38,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00347845 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:12:48,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:12:48,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155285 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:12:48,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:12:48,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813149 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:12:58,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:12:58,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101886 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:12:58,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:12:58,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00182205 s, 2.2 MB/s\n'; <rc> = 0
>Thread-14004::WARNING::2013-03-17 09:13:01,074::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:13:01,077::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 09:13:08,523::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 09:13:08,524::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 09:13:08,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:13:08,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:13:08,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104049 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:13:08,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0022396 s, 914 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:13:08,638::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0022396 s, 914 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 09:13:08,641::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 09:13:08,643::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 09:13:08,647::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 09:13:08,659::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:13:08,662::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:13:08,664::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:13:08,667::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:13:08,669::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:13:08,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:13:09,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:13:09,114::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:13:09,115::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 09:13:09,117::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 09:13:09,118::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 09:13:09,119::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 09:13:09,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:13:09,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:13:09,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:13:09,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00396603 s, 1.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:13:11,080::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:13:11,081::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:13:11,508::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:13:11,516::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:13:11,517::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:13:18,750::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 09:13:18,753::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 09:13:18,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:13:18,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000743728 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:13:18,801::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000743728 s, 2.8 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 09:13:18,801::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 09:13:18,802::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 09:13:18,803::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 09:13:18,804::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:13:18,805::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:13:18,806::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:13:18,806::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:13:18,806::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:13:18,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:13:19,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:13:19,253::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:13:19,254::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 09:13:19,256::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 09:13:19,258::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 09:13:19,258::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 09:13:19,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:13:19,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:13:19,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00341445 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:13:19,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:13:19,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:13:19,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118449 s, 346 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:13:29,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:13:29,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831994 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:13:29,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:13:29,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855669 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:13:39,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:13:39,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143763 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:13:39,875::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:13:39,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086882 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:13:49,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:13:49,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132875 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:13:49,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:13:49,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108725 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:13:59,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:13:59,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153843 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:14:00,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:14:00,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782018 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:14:09,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:14:09,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103301 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:14:10,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:14:10,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00796252 s, 514 kB/s\n'; <rc> = 0
>Thread-14005::WARNING::2013-03-17 09:14:11,544::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:14:11,547::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 09:14:19,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:14:19,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803706 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:14:20,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:14:20,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841611 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:14:21,549::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:14:21,551::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:14:21,992::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:14:21,999::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:14:22,001::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:14:29,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:14:29,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000910024 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:14:30,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:14:30,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776687 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:14:39,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:14:39,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000905727 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:14:40,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:14:40,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880087 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:14:49,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:14:49,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884441 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:14:50,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:14:50,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814626 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:14:59,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:14:59,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165044 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:15:00,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:15:00,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010187 s, 402 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:15:10,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:15:10,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138697 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:15:11,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:15:11,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159358 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:15:20,048::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:15:20,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153625 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:15:21,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:15:21,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122533 s, 3.3 MB/s\n'; <rc> = 0
>Thread-14006::WARNING::2013-03-17 09:15:22,030::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:15:22,036::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 09:15:30,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:15:30,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814451 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:15:31,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:15:31,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817273 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:15:32,038::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 09:15:32,040::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:15:32,041::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:15:32,501::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:15:32,508::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:15:32,509::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:15:40,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:15:40,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010808 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:15:41,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:15:41,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834141 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:15:50,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:15:50,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866491 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:15:51,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:15:51,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844033 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:16:00,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:16:00,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086238 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:16:01,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:16:01,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0232386 s, 176 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:16:10,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:16:10,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139839 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:16:11,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:16:11,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00071491 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:16:20,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:16:20,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000647325 s, 6.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:16:21,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:16:21,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144873 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:16:30,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:16:30,366::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000733824 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:16:32,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:16:32,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143848 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14007::WARNING::2013-03-17 09:16:32,539::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:16:32,545::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 09:16:40,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:16:40,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000989333 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:16:42,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:16:42,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846997 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:16:42,548::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:16:42,550::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:16:42,997::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:16:43,004::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:16:43,005::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:16:50,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:16:50,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000930025 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:16:52,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:16:52,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788726 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:17:00,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:17:00,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762391 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:17:02,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:17:02,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836451 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:17:10,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:17:10,526::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165948 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:17:12,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:17:12,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851342 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:17:20,532::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:17:20,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168953 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:17:22,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:17:22,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165381 s, 248 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:17:30,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:17:30,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00093862 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:17:32,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:17:32,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159961 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:17:40,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:17:40,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140463 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:17:42,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:17:42,805::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00316073 s, 1.3 MB/s\n'; <rc> = 0
>Thread-14008::WARNING::2013-03-17 09:17:43,032::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:17:43,042::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 09:17:50,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:17:50,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078664 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:17:53,045::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:17:53,047::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:17:53,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:17:53,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117065 s, 350 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:17:53,532::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:17:53,537::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:17:53,538::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:18:00,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:18:00,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822591 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:18:03,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:18:03,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819505 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:18:10,745::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 09:18:10,747::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 09:18:10,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:18:10,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00122717 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:18:10,785::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00122717 s, 1.7 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 09:18:10,786::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 09:18:10,786::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 09:18:10,788::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 09:18:10,789::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:18:10,789::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:18:10,790::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:18:10,790::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:18:10,790::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:18:10,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:18:11,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:18:11,233::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:18:11,234::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 09:18:11,236::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 09:18:11,238::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 09:18:11,238::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 09:18:11,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:18:11,607::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:18:11,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:18:11,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015995 s, 256 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:18:13,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:18:13,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168904 s, 243 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:18:21,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:18:21,692::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132817 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:18:23,391::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 09:18:23,393::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 09:18:23,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:18:23,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00136367 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:18:23,444::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00136367 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 09:18:23,445::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 09:18:23,445::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 09:18:23,447::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 09:18:23,448::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:18:23,448::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:18:23,449::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:18:23,449::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:18:23,449::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:18:23,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:18:23,866::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:18:23,870::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:18:23,871::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 09:18:23,874::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 09:18:23,875::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 09:18:23,876::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 09:18:23,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:18:24,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:18:24,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:18:24,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01307 s, 313 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:18:31,697::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:18:31,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146286 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:18:34,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:18:34,401::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841187 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:18:41,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:18:41,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856656 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:18:44,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:18:44,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112145 s, 365 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:18:51,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:18:51,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111936 s, 3.7 MB/s\n'; <rc> = 0
>Thread-14009::WARNING::2013-03-17 09:18:53,566::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:18:53,570::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:18:54,537::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:18:54,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100125 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:19:01,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:19:01,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818128 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:19:03,573::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:19:03,574::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:19:04,013::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:19:04,020::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:19:04,021::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:19:04,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:19:04,649::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156621 s, 262 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:19:11,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:19:11,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808592 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:19:14,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:19:14,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164709 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:19:21,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:19:21,958::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149591 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:19:24,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:19:24,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108346 s, 378 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:19:31,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:19:32,010::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149046 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:19:34,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:19:34,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100681 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:19:42,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:19:42,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168643 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:19:44,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:19:44,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163757 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:19:52,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:19:52,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774026 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:19:55,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:19:55,040::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860654 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:20:02,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:20:02,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130608 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14010::WARNING::2013-03-17 09:20:04,064::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:20:04,066::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:20:05,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:20:05,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885327 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:20:12,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:20:12,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738609 s, 5.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:20:14,071::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:20:14,073::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:20:14,520::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:20:14,528::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:20:14,529::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:20:15,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:20:15,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110299 s, 371 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:20:22,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:20:22,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155156 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:20:25,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:20:25,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143966 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:20:32,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:20:32,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785255 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:20:35,324::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:20:35,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754799 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:20:42,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:20:42,311::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000934206 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:20:45,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:20:45,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158556 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:20:52,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:20:52,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143725 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:20:55,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:20:55,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122475 s, 334 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:21:02,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:21:02,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143084 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:21:05,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:21:05,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00215791 s, 1.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:21:12,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:21:12,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126968 s, 3.2 MB/s\n'; <rc> = 0
>Thread-14011::WARNING::2013-03-17 09:21:14,557::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:21:14,561::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:21:15,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:21:15,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.021109 s, 194 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:21:22,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:21:22,491::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768772 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:21:24,562::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 09:21:24,563::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:21:24,564::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:21:25,018::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:21:25,024::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:21:25,026::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:21:25,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:21:25,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00800796 s, 511 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:21:32,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:21:32,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140124 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:21:35,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:21:35,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00957305 s, 428 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:21:42,537::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:21:42,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145836 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:21:45,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:21:45,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861407 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:21:52,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:21:52,631::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145876 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:21:56,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:21:56,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00519651 s, 788 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:22:02,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:22:02,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134868 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:22:06,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:22:06,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000701988 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:22:12,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:22:12,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132715 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:22:16,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:22:16,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132871 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:22:22,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:22:22,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107582 s, 3.8 MB/s\n'; <rc> = 0
>Thread-14012::WARNING::2013-03-17 09:22:25,052::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:22:25,059::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:22:26,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:22:26,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00995545 s, 411 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:22:32,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:22:32,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105679 s, 3.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:22:35,063::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:22:35,065::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:22:35,527::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:22:35,534::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:22:35,535::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:22:36,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:22:36,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112955 s, 363 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:22:42,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:22:42,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00444166 s, 922 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:22:46,415::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:22:46,448::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116068 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:22:52,859::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:22:52,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803708 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:22:56,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:22:56,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0082063 s, 499 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:23:02,894::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:23:02,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163335 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:23:06,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:23:06,633::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153546 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:23:12,958::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 09:23:12,959::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 09:23:12,959::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:23:12,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00132219 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:23:12,997::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00132219 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 09:23:12,998::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 09:23:12,998::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 09:23:13,000::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 09:23:13,002::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:23:13,003::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:23:13,003::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:23:13,003::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:23:13,004::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:23:13,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:23:13,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:23:13,425::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:23:13,427::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 09:23:13,429::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 09:23:13,430::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 09:23:13,430::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 09:23:13,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:23:13,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:23:13,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:23:13,818::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102777 s, 399 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:23:16,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:23:16,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011719 s, 350 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:23:23,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:23:23,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794716 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:23:26,768::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 09:23:26,770::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 09:23:26,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:23:26,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00127971 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:23:26,811::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00127971 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 09:23:26,811::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 09:23:26,812::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 09:23:26,813::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 09:23:26,815::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:23:26,815::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:23:26,816::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:23:26,817::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:23:26,817::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:23:26,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:23:27,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:23:27,260::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:23:27,261::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 09:23:27,264::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 09:23:27,265::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 09:23:27,265::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 09:23:27,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:23:27,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:23:27,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:23:27,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124643 s, 329 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:23:33,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:23:33,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114767 s, 3.6 MB/s\n'; <rc> = 0
>Thread-14013::WARNING::2013-03-17 09:23:35,566::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:23:35,572::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:23:37,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:23:37,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163849 s, 250 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:23:43,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:23:43,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920816 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:23:45,574::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:23:45,576::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:23:46,063::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:23:46,071::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:23:46,073::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:23:47,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:23:47,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013795 s, 297 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:23:53,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:23:53,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158431 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:23:57,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:23:58,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119659 s, 342 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:24:03,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:24:04,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00953215 s, 430 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:24:08,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:24:08,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773225 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:24:14,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:24:14,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00257619 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:24:18,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:24:18,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187068 s, 219 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:24:24,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:24:24,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103421 s, 396 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:24:28,194::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:24:28,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166616 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:24:34,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:24:34,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133514 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:24:38,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:24:38,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124283 s, 330 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:24:44,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:24:44,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116825 s, 351 kB/s\n'; <rc> = 0
>Thread-14014::WARNING::2013-03-17 09:24:46,099::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:24:46,103::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:24:48,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:24:48,395::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000745875 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:24:54,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:24:54,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841497 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:24:56,109::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:24:56,110::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:24:56,568::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:24:56,574::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:24:56,576::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:24:58,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:24:58,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00780022 s, 525 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:25:04,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:25:04,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000727082 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:25:08,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:25:08,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147818 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:25:14,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:25:14,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137328 s, 298 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:25:18,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:25:18,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920549 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:25:24,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:25:24,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115269 s, 355 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:25:28,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:25:28,733::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134178 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:25:34,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:25:34,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00279279 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:25:38,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:25:38,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155913 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:25:44,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:25:44,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142436 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:25:48,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:25:48,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900228 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:25:54,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:25:54,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102541 s, 4.0 MB/s\n'; <rc> = 0
>Thread-14015::WARNING::2013-03-17 09:25:56,600::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:25:56,603::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:25:58,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:25:58,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123336 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:26:04,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:26:04,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816026 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:26:06,610::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:26:06,611::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:26:07,058::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:26:07,065::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:26:07,066::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:26:09,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:26:09,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146626 s, 279 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:26:14,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:26:14,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868647 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:26:19,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:26:19,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844581 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:26:24,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:26:24,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085426 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:26:29,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:26:29,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101786 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:26:34,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:26:34,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103694 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:26:39,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:26:39,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760149 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:26:44,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:26:44,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895741 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:26:49,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:26:49,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000914904 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:26:54,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:26:54,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181992 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:26:59,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:26:59,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145683 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:27:04,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:27:04,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786506 s, 5.2 MB/s\n'; <rc> = 0
>Thread-14016::WARNING::2013-03-17 09:27:07,090::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:27:07,094::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:27:09,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:27:09,484::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832485 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:27:14,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:27:14,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815435 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:27:17,149::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 09:27:17,150::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:27:17,151::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:27:17,575::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:27:17,581::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:27:17,583::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:27:19,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:27:19,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148599 s, 276 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:27:24,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:27:24,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000991446 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:27:29,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:27:29,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174738 s, 234 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:27:34,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:27:34,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131542 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:27:39,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:27:39,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894037 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:27:44,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:27:45,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148842 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:27:49,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:27:49,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008363 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:27:55,048::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:27:55,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000708553 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:27:59,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:27:59,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138365 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:28:05,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:28:05,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0038934 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:28:09,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:28:09,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891387 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:28:15,158::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 09:28:15,159::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 09:28:15,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:28:15,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0013729 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:28:15,207::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0013729 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 09:28:15,208::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 09:28:15,208::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 09:28:15,210::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 09:28:15,211::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:28:15,211::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:28:15,212::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:28:15,212::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:28:15,212::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:28:15,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:28:15,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:28:15,669::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:28:15,670::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 09:28:15,673::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 09:28:15,674::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 09:28:15,675::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 09:28:15,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:28:16,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:28:16,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:28:16,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0220911 s, 185 kB/s\n'; <rc> = 0
>Thread-14017::WARNING::2013-03-17 09:28:17,614::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:28:17,620::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:28:20,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:28:20,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01295 s, 316 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:28:26,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:28:26,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178467 s, 230 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:28:27,623::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:28:27,624::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:28:28,035::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:28:28,041::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:28:28,043::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:28:30,133::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 09:28:30,135::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 09:28:30,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:28:30,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0147063 s, 139 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:28:30,180::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0147063 s, 139 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 09:28:30,180::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 09:28:30,181::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 09:28:30,182::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 09:28:30,186::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:28:30,187::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:28:30,187::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:28:30,188::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:28:30,188::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:28:30,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:28:30,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:28:30,559::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:28:30,560::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 09:28:30,563::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 09:28:30,564::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 09:28:30,564::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 09:28:30,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:28:30,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:28:30,922::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:28:30,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00403136 s, 1.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:28:36,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:28:36,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851334 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:28:40,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:28:41,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791391 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:28:46,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:28:46,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00983437 s, 416 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:28:51,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:28:51,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813926 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:28:56,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:28:56,409::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792108 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:29:01,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:29:01,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749675 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:29:06,416::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:29:06,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894502 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:29:11,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:29:11,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00264897 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:29:16,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:29:16,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771785 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:29:21,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:29:21,333::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157216 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:29:26,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:29:26,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000954913 s, 4.3 MB/s\n'; <rc> = 0
>Thread-14018::WARNING::2013-03-17 09:29:28,070::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:29:28,073::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:29:31,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:29:31,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771038 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:29:36,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:29:36,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825126 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:29:38,080::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:29:38,081::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:29:38,526::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:29:38,533::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:29:38,535::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:29:41,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:29:41,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0201948 s, 203 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:29:46,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:29:46,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016923 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:29:51,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:29:51,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147371 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:29:56,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:29:56,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854281 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:30:01,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:30:01,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00307361 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:30:06,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:30:06,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0071507 s, 573 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:30:11,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:30:11,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00449222 s, 912 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:30:16,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:30:16,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000998205 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:30:21,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:30:21,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133464 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:30:26,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:30:26,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00202089 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:30:31,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:30:31,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870766 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:30:36,938::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:30:36,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872101 s, 4.7 MB/s\n'; <rc> = 0
>Thread-14019::WARNING::2013-03-17 09:30:38,567::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:30:38,572::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:30:41,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:30:42,017::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114529 s, 358 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:30:46,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:30:47,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777019 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:30:48,574::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:30:48,575::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:30:49,007::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:30:49,014::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:30:49,015::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:30:52,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:30:52,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191042 s, 214 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:30:57,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:30:57,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791973 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:31:02,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:31:02,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876193 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:31:07,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:31:07,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848686 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:31:12,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:31:12,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0220015 s, 186 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:31:17,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:31:17,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860132 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:31:22,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:31:22,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765293 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:31:27,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:31:27,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137897 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:31:32,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:31:32,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0196897 s, 208 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:31:37,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:31:37,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147089 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:31:42,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:31:42,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161831 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:31:47,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:31:47,266::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107685 s, 3.8 MB/s\n'; <rc> = 0
>Thread-14020::WARNING::2013-03-17 09:31:49,055::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:31:49,057::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:31:52,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:31:52,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000934136 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:31:57,280::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:31:57,309::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849013 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:31:59,067::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:31:59,069::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:31:59,473::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:31:59,479::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:31:59,481::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:32:02,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:32:02,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138047 s, 297 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:32:07,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:32:07,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00521017 s, 786 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:32:12,741::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:32:12,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080973 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:32:17,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:32:17,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000969206 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:32:22,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:32:22,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00318985 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:32:27,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:32:27,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827148 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:32:32,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:32:32,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110747 s, 370 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:32:37,439::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:32:37,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171608 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:32:42,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:32:43,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788821 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:32:47,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:32:47,511::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764296 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:32:53,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:32:53,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860902 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:32:57,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:32:57,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000981965 s, 4.2 MB/s\n'; <rc> = 0
>Thread-14021::WARNING::2013-03-17 09:32:59,510::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:32:59,514::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:33:03,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:33:03,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081973 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:33:07,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:33:07,593::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00073608 s, 5.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:33:09,515::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 09:33:09,516::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:33:09,517::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:33:09,949::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:33:09,956::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:33:09,957::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:33:13,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:33:13,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159878 s, 256 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:33:17,598::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 09:33:17,599::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 09:33:17,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:33:17,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000885679 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:33:17,630::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000885679 s, 2.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 09:33:17,630::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 09:33:17,630::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 09:33:17,632::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 09:33:17,633::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:33:17,634::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:33:17,634::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:33:17,635::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:33:17,635::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:33:17,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:33:18,054::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:33:18,059::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:33:18,060::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 09:33:18,062::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 09:33:18,064::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 09:33:18,064::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 09:33:18,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:33:18,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:33:18,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:33:18,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160381 s, 255 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:33:23,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:33:23,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0244801 s, 167 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:33:28,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:33:28,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135572 s, 302 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:33:33,403::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 09:33:33,404::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 09:33:33,405::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:33:33,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00126469 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:33:33,447::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00126469 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 09:33:33,448::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 09:33:33,448::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 09:33:33,450::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 09:33:33,451::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:33:33,451::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:33:33,452::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:33:33,452::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:33:33,453::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:33:33,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:33:33,890::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:33:33,895::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:33:33,896::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 09:33:33,899::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 09:33:33,900::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 09:33:33,901::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 09:33:33,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:33:34,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:33:34,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:33:34,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112711 s, 363 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:33:38,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:33:38,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826585 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:33:44,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:33:44,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765649 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:33:48,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:33:48,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133328 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:33:54,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:33:54,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803812 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:33:58,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:33:58,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154709 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:34:04,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:34:04,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000996623 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:34:08,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:34:08,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000676016 s, 6.1 MB/s\n'; <rc> = 0
>Thread-14022::WARNING::2013-03-17 09:34:09,985::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:34:09,989::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:34:14,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:34:14,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808636 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:34:18,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:34:18,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837113 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:34:19,992::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:34:19,994::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:34:20,434::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:34:20,440::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:34:20,442::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:34:24,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:34:24,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00906982 s, 452 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:34:28,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:34:28,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144458 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:34:34,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:34:34,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000938298 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:34:38,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:34:38,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850688 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:34:44,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:34:44,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819321 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:34:48,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:34:48,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000991947 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:34:54,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:34:54,907::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150154 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:34:58,944::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:34:58,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155622 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:35:04,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:35:04,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138542 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:35:08,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:35:09,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137044 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:35:15,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:35:15,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140506 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:35:19,070::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:35:19,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00217347 s, 1.9 MB/s\n'; <rc> = 0
>Thread-14023::WARNING::2013-03-17 09:35:20,469::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:35:20,475::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:35:25,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:35:25,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728939 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:35:29,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:35:29,191::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079819 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:35:30,484::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:35:30,485::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:35:30,978::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:35:30,986::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:35:30,988::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:35:35,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:35:35,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166924 s, 245 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:35:39,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:35:39,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900539 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:35:45,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:35:45,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164019 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:35:49,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:35:49,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000971161 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:35:55,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:35:55,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783113 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:35:59,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:35:59,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782753 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:36:05,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:36:05,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778047 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:36:09,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:36:09,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145396 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:36:15,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:36:15,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178502 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:36:19,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:36:19,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831759 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:36:25,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:36:25,627::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00290795 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:36:29,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:36:29,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091293 s, 4.5 MB/s\n'; <rc> = 0
>Thread-14024::WARNING::2013-03-17 09:36:31,012::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:36:31,016::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:36:35,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:36:35,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000999081 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:36:39,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:36:39,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081733 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:36:41,018::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:36:41,020::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:36:41,437::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:36:41,444::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:36:41,445::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:36:45,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:36:45,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00799055 s, 513 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:36:49,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:36:49,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901378 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:36:55,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:36:55,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131514 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:36:59,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:36:59,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148838 s, 275 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:37:05,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:37:05,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139123 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:37:09,681::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:37:09,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000987372 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:37:15,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:37:16,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826323 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:37:19,717::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:37:19,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772591 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:37:26,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:37:26,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000949465 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:37:29,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:37:29,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000928214 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:37:36,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:37:36,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131218 s, 312 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:37:39,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:37:39,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134383 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14025::WARNING::2013-03-17 09:37:41,470::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:37:41,477::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:37:46,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:37:46,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818483 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:37:49,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:37:49,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785839 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:37:51,480::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:37:51,481::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:37:51,892::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:37:51,899::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:37:51,900::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:37:56,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:37:56,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00703615 s, 582 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:37:59,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:37:59,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144283 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:38:06,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:38:06,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899498 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:38:09,941::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:38:09,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814312 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:38:16,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:38:16,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000993886 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:38:19,972::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 09:38:19,974::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 09:38:19,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:38:20,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000842284 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:38:20,003::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000842284 s, 2.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 09:38:20,003::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 09:38:20,004::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 09:38:20,006::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 09:38:20,007::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:38:20,007::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:38:20,008::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:38:20,008::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:38:20,008::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:38:20,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:38:20,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:38:20,440::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:38:20,442::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 09:38:20,445::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 09:38:20,446::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 09:38:20,446::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 09:38:20,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:38:20,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:38:20,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:38:20,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00677289 s, 605 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:38:26,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:38:26,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124652 s, 329 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:38:30,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:38:30,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178798 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:38:36,623::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 09:38:36,625::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 09:38:36,625::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:38:36,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00154227 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:38:36,666::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00154227 s, 1.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 09:38:36,666::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 09:38:36,667::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 09:38:36,668::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 09:38:36,669::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:38:36,670::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:38:36,671::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:38:36,671::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:38:36,671::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:38:36,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:38:37,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:38:37,142::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:38:37,147::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 09:38:37,149::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 09:38:37,151::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 09:38:37,151::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 09:38:37,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:38:37,517::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:38:37,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:38:37,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00942481 s, 435 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:38:40,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:38:40,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892395 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:38:47,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:38:47,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809107 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:38:51,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:38:51,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829477 s, 4.9 MB/s\n'; <rc> = 0
>Thread-14026::WARNING::2013-03-17 09:38:51,928::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:38:51,932::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:38:57,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:38:57,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819216 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:39:01,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:39:01,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800639 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:39:01,934::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 09:39:01,936::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:39:01,937::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:39:02,394::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:39:02,401::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:39:02,402::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:39:07,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:39:07,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145552 s, 281 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:39:11,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:39:11,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00259029 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:39:17,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:39:17,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872832 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:39:21,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:39:21,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000713873 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:39:27,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:39:27,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155735 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:39:31,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:39:31,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165144 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:39:38,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:39:38,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00090057 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:39:41,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:39:41,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136908 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:39:48,070::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:39:48,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830365 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:39:51,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:39:51,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140251 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:39:58,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:39:58,188::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150686 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:40:01,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:40:01,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0286756 s, 143 kB/s\n'; <rc> = 0
>Thread-14027::WARNING::2013-03-17 09:40:02,434::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:40:02,440::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:40:08,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:40:08,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00240467 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:40:11,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:40:11,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148455 s, 276 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:40:12,453::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:40:12,454::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:40:12,907::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:40:12,914::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:40:12,916::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:40:18,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:40:18,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014548 s, 282 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:40:21,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:40:21,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139531 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:40:28,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:40:28,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017074 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:40:31,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:40:31,809::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737574 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:40:38,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:40:38,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757236 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:40:41,814::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:40:41,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137239 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:40:48,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:40:48,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168608 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:40:51,859::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:40:51,890::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775497 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:40:58,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:40:58,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00223632 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:41:01,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:41:01,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797223 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:41:08,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:41:08,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144356 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:41:11,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:41:12,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0210171 s, 195 kB/s\n'; <rc> = 0
>Thread-14028::WARNING::2013-03-17 09:41:12,946::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:41:12,950::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:41:18,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:41:18,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812932 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:41:22,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:41:22,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833901 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:41:22,952::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:41:22,954::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:41:23,411::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:41:23,418::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:41:23,419::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:41:28,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:41:28,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179998 s, 228 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:41:32,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:41:32,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0272683 s, 150 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:41:38,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:41:38,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797768 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:41:42,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:41:42,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828679 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:41:49,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:41:49,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135378 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:41:52,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:41:52,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108428 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:41:59,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:41:59,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795041 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:42:02,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:42:02,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138063 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:42:09,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:42:09,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769239 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:42:12,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:42:12,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135176 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:42:19,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:42:19,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100277 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:42:22,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:42:22,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00398407 s, 1.0 MB/s\n'; <rc> = 0
>Thread-14029::WARNING::2013-03-17 09:42:23,452::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:42:23,455::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:42:29,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:42:29,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796968 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:42:32,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:42:32,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078205 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:42:33,457::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:42:33,459::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:42:33,892::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:42:33,899::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:42:33,901::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:42:39,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:42:39,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00832356 s, 492 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:42:42,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:42:42,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101192 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:42:49,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:42:49,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779669 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:42:52,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:42:52,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00093193 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:42:59,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:42:59,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138468 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:43:02,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:43:02,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00073189 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:43:09,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:43:09,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789661 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:43:12,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:43:12,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152892 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:43:19,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:43:19,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140421 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:43:22,643::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 09:43:22,645::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 09:43:22,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:43:22,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00145595 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:43:22,686::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00145595 s, 1.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 09:43:22,687::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 09:43:22,687::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 09:43:22,689::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 09:43:22,690::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:43:22,690::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:43:22,691::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:43:22,691::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:43:22,692::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:43:22,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:43:23,132::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:43:23,136::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:43:23,137::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 09:43:23,140::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 09:43:23,141::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 09:43:23,141::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 09:43:23,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:43:23,517::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:43:23,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:43:23,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137727 s, 297 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:43:29,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:43:29,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00565072 s, 725 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:43:33,571::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:43:33,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00482737 s, 848 kB/s\n'; <rc> = 0
>Thread-14030::WARNING::2013-03-17 09:43:33,936::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:43:33,940::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:43:39,917::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 09:43:39,918::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 09:43:39,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:43:39,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000854959 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:43:39,947::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000854959 s, 2.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 09:43:39,947::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 09:43:39,947::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 09:43:39,949::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 09:43:39,950::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:43:39,951::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:43:39,951::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:43:39,952::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:43:39,952::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:43:39,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:43:40,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:43:40,362::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:43:40,363::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 09:43:40,365::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 09:43:40,366::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 09:43:40,367::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 09:43:40,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:43:40,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:43:40,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:43:40,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00288104 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:43:43,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:43:43,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887491 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:43:43,943::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:43:43,945::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:43:44,406::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:43:44,413::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:43:44,414::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:43:50,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:43:50,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00812309 s, 504 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:43:53,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:43:53,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087276 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:44:00,863::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:44:00,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911869 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:44:03,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:44:03,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917126 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:44:10,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:44:10,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00852556 s, 480 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:44:13,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:44:13,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896517 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:44:21,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:44:21,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176818 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:44:23,796::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:44:23,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797847 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:44:31,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:44:31,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014255 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:44:33,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:44:33,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157049 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:44:41,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:44:41,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164239 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:44:43,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:44:43,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00333685 s, 1.2 MB/s\n'; <rc> = 0
>Thread-14031::WARNING::2013-03-17 09:44:44,449::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:44:44,453::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:44:51,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:44:51,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828013 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:44:53,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:44:54,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078375 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:44:54,509::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 09:44:54,518::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:44:54,520::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:44:54,941::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:44:54,948::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:44:54,950::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:45:01,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:45:01,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150719 s, 272 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:45:04,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:45:04,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748093 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:45:11,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:45:11,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851623 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:45:14,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:45:14,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00273296 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:45:21,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:45:21,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840672 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:45:24,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:45:24,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000727052 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:45:31,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:45:31,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012998 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:45:34,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:45:34,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810615 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:45:41,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:45:41,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016451 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:45:44,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:45:44,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137362 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:45:51,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:45:51,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146137 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:45:54,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:45:54,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836439 s, 4.9 MB/s\n'; <rc> = 0
>Thread-14032::WARNING::2013-03-17 09:45:54,974::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:45:54,978::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:46:01,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:46:01,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008325 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:46:04,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:46:04,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00183352 s, 2.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:46:04,980::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:46:04,982::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:46:05,398::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:46:05,405::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:46:05,407::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:46:11,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:46:11,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104538 s, 392 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:46:14,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:46:14,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181639 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:46:21,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:46:22,017::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178789 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:46:24,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:46:24,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137095 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:46:32,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:46:32,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083241 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:46:34,648::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:46:34,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013021 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:46:42,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:46:42,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144963 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:46:44,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:46:44,742::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148587 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:46:52,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:46:52,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155167 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:46:54,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:46:54,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815081 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:47:02,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:47:02,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804244 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:47:04,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:47:04,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898902 s, 4.6 MB/s\n'; <rc> = 0
>Thread-14033::WARNING::2013-03-17 09:47:05,447::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:47:05,450::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:47:12,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:47:12,392::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157798 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:47:14,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:47:15,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741865 s, 5.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:47:15,453::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:47:15,455::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:47:15,902::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:47:15,909::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:47:15,911::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:47:22,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:47:22,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112 s, 366 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:47:25,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:47:25,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170163 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:47:32,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:47:32,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805421 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:47:35,070::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:47:35,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812463 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:47:42,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:47:42,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137005 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:47:45,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:47:45,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148927 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:47:52,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:47:52,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761288 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:47:55,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:47:55,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139813 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:48:02,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:48:02,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783161 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:48:05,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:48:05,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741297 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:48:12,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:48:12,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145181 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:48:15,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:48:15,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143848 s, 285 kB/s\n'; <rc> = 0
>Thread-14034::WARNING::2013-03-17 09:48:15,938::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:48:15,942::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:48:22,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:48:22,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839713 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:48:25,410::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 09:48:25,411::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 09:48:25,412::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:48:25,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00148365 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:48:25,468::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00148365 s, 1.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 09:48:25,468::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 09:48:25,468::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 09:48:25,470::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 09:48:25,471::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:48:25,473::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:48:25,473::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:48:25,474::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:48:25,474::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:48:25,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:48:25,953::lvm::373::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-21::DEBUG::2013-03-17 09:48:25,976::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:48:26,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:48:26,048::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 09:48:26,051::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 09:48:26,052::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 09:48:26,053::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 09:48:26,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:48:26,451::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-22::DEBUG::2013-03-17 09:48:26,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:48:26,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-21::WARNING::2013-03-17 09:48:26,530::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:48:26,541::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:48:26,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01395 s, 294 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:48:32,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:48:32,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00914807 s, 448 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:48:36,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:48:36,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794561 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:48:43,034::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 09:48:43,036::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 09:48:43,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:48:43,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000713356 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:48:43,068::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000713356 s, 2.9 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 09:48:43,068::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 09:48:43,068::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 09:48:43,070::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 09:48:43,071::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:48:43,072::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:48:43,072::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:48:43,072::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:48:43,073::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:48:43,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:48:43,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:48:43,492::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:48:43,493::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 09:48:43,497::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 09:48:43,498::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 09:48:43,499::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 09:48:43,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:48:43,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:48:43,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:48:43,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00967438 s, 423 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:48:46,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:48:46,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129345 s, 317 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:48:53,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:48:54,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0043909 s, 933 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:48:56,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:48:56,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787456 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:49:04,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:49:04,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000889935 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:49:06,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:49:06,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831871 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:49:14,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:49:14,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016764 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:49:16,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:49:16,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141661 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:49:24,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:49:24,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101718 s, 4.0 MB/s\n'; <rc> = 0
>Thread-14035::WARNING::2013-03-17 09:49:26,585::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:49:26,588::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 09:49:26,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:49:26,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117871 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:49:34,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:49:34,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000918195 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:49:36,590::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:49:36,591::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:49:36,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:49:36,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152488 s, 269 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:49:37,077::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:49:37,083::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:49:37,085::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:49:44,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:49:44,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177089 s, 231 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:49:46,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:49:46,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782111 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:49:54,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:49:54,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780734 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:49:56,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:49:56,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130708 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:50:04,537::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:50:04,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150475 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:50:06,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:50:07,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0298833 s, 137 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:50:14,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:50:14,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136476 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:50:17,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:50:17,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0207221 s, 198 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:50:24,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:50:24,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811585 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:50:27,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:50:27,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142386 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:50:34,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:50:34,801::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012396 s, 3.3 MB/s\n'; <rc> = 0
>Thread-14036::WARNING::2013-03-17 09:50:37,114::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:50:37,119::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 09:50:37,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:50:37,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171111 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:50:44,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:50:44,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000914403 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:50:47,121::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 09:50:47,122::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:50:47,123::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:50:47,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:50:47,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199655 s, 205 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:50:47,571::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:50:47,577::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:50:47,579::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:50:54,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:50:54,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179366 s, 228 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:50:57,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:50:57,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867199 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:51:05,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:51:05,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821671 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:51:07,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:51:07,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00188986 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:51:15,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:51:15,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143769 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:51:17,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:51:17,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135039 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:51:25,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:51:25,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000651732 s, 6.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:51:27,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:51:27,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846759 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:51:35,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:51:35,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016584 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:51:37,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:51:37,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774383 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:51:45,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:51:45,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107693 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:51:47,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:51:47,513::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744001 s, 5.5 MB/s\n'; <rc> = 0
>Thread-14037::WARNING::2013-03-17 09:51:47,628::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:51:47,636::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:51:55,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:51:55,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859464 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:51:57,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:51:57,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00679192 s, 603 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:51:57,639::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:51:57,641::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:51:58,115::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:51:58,122::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:51:58,123::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:52:05,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:52:05,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015695 s, 261 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:52:07,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:52:07,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886598 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:52:15,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:52:15,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838359 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:52:17,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:52:17,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103453 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:52:25,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:52:25,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145316 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:52:27,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:52:27,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0247347 s, 166 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:52:35,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:52:35,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0157856 s, 259 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:52:37,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:52:37,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008266 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:52:45,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:52:45,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145768 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:52:47,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:52:47,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188924 s, 217 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:52:55,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:52:55,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088661 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:52:57,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:52:58,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00241862 s, 1.7 MB/s\n'; <rc> = 0
>Thread-14038::WARNING::2013-03-17 09:52:58,152::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:52:58,158::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:53:05,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:53:06,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821461 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:53:08,060::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:53:08,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135923 s, 301 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:53:08,163::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:53:08,165::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:53:08,636::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:53:08,643::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:53:08,645::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:53:16,079::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:53:16,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172891 s, 237 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:53:18,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:53:18,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132559 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:53:26,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:53:26,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856772 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:53:28,209::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 09:53:28,221::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 09:53:28,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:53:28,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00137156 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:53:28,281::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00137156 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 09:53:28,282::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 09:53:28,282::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 09:53:28,284::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 09:53:28,285::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:53:28,285::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:53:28,286::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:53:28,286::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:53:28,286::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:53:28,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:53:28,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:53:28,719::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:53:28,720::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 09:53:28,722::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 09:53:28,724::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 09:53:28,724::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 09:53:28,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:53:29,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:53:29,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:53:29,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0195257 s, 210 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:53:36,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:53:36,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00926671 s, 442 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:53:39,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:53:39,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743113 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:53:46,316::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 09:53:46,318::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 09:53:46,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:53:46,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00139528 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:53:46,363::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00139528 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 09:53:46,363::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 09:53:46,363::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 09:53:46,365::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 09:53:46,366::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:53:46,367::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:53:46,367::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:53:46,367::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:53:46,368::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:53:46,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:53:46,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:53:46,787::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:53:46,788::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 09:53:46,790::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 09:53:46,791::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 09:53:46,792::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 09:53:46,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:53:47,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:53:47,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:53:47,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129177 s, 317 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:53:49,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:53:49,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103661 s, 395 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:53:57,280::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:53:57,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157824 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:53:59,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:53:59,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001439 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:54:07,365::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:54:07,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134487 s, 305 kB/s\n'; <rc> = 0
>Thread-14039::WARNING::2013-03-17 09:54:08,673::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:54:08,678::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 09:54:09,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:54:09,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161557 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:54:17,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:54:17,899::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162611 s, 2.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:54:18,682::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:54:18,684::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:54:19,116::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:54:19,123::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:54:19,124::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:54:19,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:54:19,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176693 s, 232 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:54:27,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:54:28,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156229 s, 262 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:54:29,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:54:29,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078056 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:54:38,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:54:38,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00090318 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:54:39,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:54:39,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000931082 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:54:48,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:54:48,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00353042 s, 1.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:54:49,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:54:49,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148335 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:54:58,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:54:58,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802609 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:54:59,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:54:59,561::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155805 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:55:08,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:55:08,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907005 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:55:09,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:55:09,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780679 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:55:18,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:55:18,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828403 s, 4.9 MB/s\n'; <rc> = 0
>Thread-14040::WARNING::2013-03-17 09:55:19,152::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:55:19,162::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 09:55:19,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:55:19,700::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0252774 s, 162 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:55:28,647::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:55:28,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0322837 s, 127 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:55:29,164::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:55:29,166::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:55:29,562::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:55:29,569::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:55:29,570::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:55:29,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:55:29,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203288 s, 201 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:55:38,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:55:38,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808879 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:55:39,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:55:39,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000961405 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:55:48,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:55:48,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146946 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:55:49,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:55:49,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140957 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:55:58,910::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:55:58,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805737 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:55:59,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:55:59,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167412 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:56:08,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:56:09,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000732621 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:56:09,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:56:09,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770195 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:56:19,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:56:19,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174843 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:56:19,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:56:19,973::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000889991 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:56:29,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:56:29,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128745 s, 3.2 MB/s\n'; <rc> = 0
>Thread-14041::WARNING::2013-03-17 09:56:29,595::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:56:29,599::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 09:56:29,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:56:30,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819465 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:56:39,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:56:39,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108206 s, 379 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:56:39,601::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 09:56:39,603::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:56:39,604::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:56:40,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:56:40,050::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:56:40,060::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:56:40,062::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:56:40,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173888 s, 236 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:56:49,341::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:56:49,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766411 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:56:50,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:56:50,123::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854424 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:56:59,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:56:59,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916389 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:57:00,129::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:57:00,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868679 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:57:09,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:57:09,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144809 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:57:10,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:57:10,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805975 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:57:19,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:57:19,593::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134378 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:57:20,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:57:20,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867377 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:57:29,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:57:29,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777792 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:57:30,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:57:30,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000934221 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:57:39,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:57:39,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137784 s, 297 kB/s\n'; <rc> = 0
>Thread-14042::WARNING::2013-03-17 09:57:40,097::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:57:40,101::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 09:57:40,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:57:40,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00206844 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:57:50,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:57:50,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109605 s, 374 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:57:50,108::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:57:50,117::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:57:50,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:57:50,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181416 s, 226 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:57:50,614::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:57:50,620::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:57:50,621::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:58:00,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:58:00,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150099 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:58:00,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:58:00,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124749 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:58:10,281::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:58:10,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885541 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:58:10,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:58:10,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827778 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:58:20,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:58:20,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181274 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:58:20,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:58:20,511::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165175 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:58:30,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:58:30,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01502 s, 273 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:58:30,516::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 09:58:30,518::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 09:58:30,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:58:30,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00121709 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:58:30,582::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00121709 s, 1.7 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 09:58:30,584::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 09:58:30,584::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 09:58:30,589::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 09:58:30,596::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:58:30,599::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:58:30,602::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:58:30,605::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:58:30,607::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 09:58:30,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:58:31,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:58:31,061::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:58:31,062::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 09:58:31,064::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 09:58:31,066::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 09:58:31,066::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 09:58:31,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:58:31,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:58:31,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:58:31,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159859 s, 256 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:58:40,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:58:40,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0208427 s, 197 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:58:41,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:58:41,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165373 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14043::WARNING::2013-03-17 09:58:50,660::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 09:58:50,666::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 09:58:50,766::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 09:58:50,767::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 09:58:50,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:58:50,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0189117 s, 108 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:58:50,828::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0189117 s, 108 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 09:58:50,829::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 09:58:50,829::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 09:58:50,831::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 09:58:50,832::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:58:50,832::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:58:50,835::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:58:50,836::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:58:50,836::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 09:58:50,837::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:58:51,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:58:51,257::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 09:58:51,258::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 09:58:51,260::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 09:58:51,262::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 09:58:51,262::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 09:58:51,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:58:51,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:58:51,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175281 s, 234 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:58:51,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:58:51,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:58:51,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00860372 s, 476 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 09:59:00,670::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 09:59:00,671::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 09:59:01,090::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 09:59:01,097::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 09:59:01,099::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 09:59:01,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:59:01,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154396 s, 265 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:59:01,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:59:01,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161633 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:59:11,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:59:11,712::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145209 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:59:11,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:59:11,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797362 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:59:21,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:59:21,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819223 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:59:21,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:59:22,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145833 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:59:31,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:59:31,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000713613 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:59:32,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:59:32,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848211 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:59:41,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:59:41,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133383 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:59:42,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:59:42,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792195 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 09:59:51,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 09:59:51,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.003402 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 09:59:52,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 09:59:52,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145692 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14044::WARNING::2013-03-17 10:00:01,122::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:00:01,126::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:00:02,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:00:02,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876146 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:00:02,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:00:02,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839543 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:00:11,128::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:00:11,129::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:00:11,556::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:00:11,563::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:00:11,565::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:00:12,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:00:12,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117094 s, 350 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:00:12,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:00:12,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170548 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:00:22,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:00:22,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896178 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:00:22,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:00:22,484::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00385431 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:00:32,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:00:32,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000688859 s, 5.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:00:32,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:00:32,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155315 s, 264 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:00:42,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:00:42,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080584 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:00:42,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:00:42,657::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113695 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:00:52,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:00:52,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136481 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:00:52,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:00:52,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177643 s, 231 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:01:02,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:01:02,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000941746 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:01:02,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:01:02,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138634 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14045::WARNING::2013-03-17 10:01:11,590::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:01:11,593::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:01:12,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:01:12,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777469 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:01:12,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:01:12,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00514977 s, 795 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:01:21,595::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:01:21,597::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:01:22,029::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:01:22,035::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:01:22,036::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:01:22,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:01:22,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177174 s, 231 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:01:22,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:01:23,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156281 s, 262 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:01:32,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:01:32,439::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000735338 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:01:33,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:01:33,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761017 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:01:42,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:01:42,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840148 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:01:43,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:01:43,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015494 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:01:52,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:01:52,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843174 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:01:53,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:01:53,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00433621 s, 945 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:02:02,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:02:02,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000733209 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:02:03,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:02:03,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142417 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:02:12,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:02:12,593::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796649 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:02:13,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:02:13,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155566 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14046::WARNING::2013-03-17 10:02:22,065::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:02:22,069::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:02:22,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:02:22,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103185 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:02:23,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:02:23,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872365 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:02:32,070::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 10:02:32,072::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:02:32,073::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:02:32,501::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:02:32,508::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:02:32,509::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:02:32,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:02:32,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167814 s, 244 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:02:33,560::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:02:33,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135664 s, 302 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:02:42,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:02:42,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137462 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:02:43,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:02:43,712::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140847 s, 291 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:02:52,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:02:52,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088085 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:02:53,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:02:53,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831225 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:03:02,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:03:02,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00203225 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:03:03,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:03:03,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00186933 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:03:12,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:03:12,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836953 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:03:13,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:03:13,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764993 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:03:22,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:03:22,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859067 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:03:23,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:03:24,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014622 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14047::WARNING::2013-03-17 10:03:32,535::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:03:32,538::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:03:32,901::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 10:03:32,902::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 10:03:32,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:03:32,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000816883 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:03:32,928::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000816883 s, 2.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 10:03:32,928::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 10:03:32,929::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 10:03:32,930::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 10:03:32,931::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:03:32,932::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:03:32,933::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:03:32,933::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:03:32,933::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:03:32,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:03:33,333::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:03:33,337::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:03:33,338::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 10:03:33,341::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 10:03:33,343::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 10:03:33,343::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 10:03:33,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:03:33,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:03:33,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:03:33,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015741 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:03:34,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:03:34,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145086 s, 282 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:03:42,540::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:03:42,542::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:03:43,023::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:03:43,034::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:03:43,035::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:03:43,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:03:43,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128042 s, 320 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:03:44,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:03:44,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016441 s, 249 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:03:53,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:03:53,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115229 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:03:54,237::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 10:03:54,238::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 10:03:54,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:03:54,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000781179 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:03:54,265::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000781179 s, 2.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 10:03:54,266::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 10:03:54,266::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 10:03:54,268::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 10:03:54,269::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:03:54,269::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:03:54,270::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:03:54,270::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:03:54,270::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:03:54,271::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:03:54,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:03:54,687::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:03:54,688::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 10:03:54,690::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 10:03:54,691::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 10:03:54,692::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 10:03:54,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:03:55,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:03:55,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:03:55,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00907614 s, 451 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:04:03,837::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:04:03,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847524 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:04:05,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:04:05,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782389 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:04:13,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:04:13,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143846 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:04:15,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:04:15,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765931 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:04:23,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:04:23,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137212 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:04:25,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:04:25,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000735432 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:04:33,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:04:34,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147024 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:04:35,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:04:35,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116971 s, 3.5 MB/s\n'; <rc> = 0
>Thread-14048::WARNING::2013-03-17 10:04:43,061::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:04:43,064::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:04:44,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:04:44,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00976385 s, 420 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:04:45,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:04:45,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078243 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:04:53,074::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:04:53,075::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:04:53,478::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:04:53,484::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:04:53,486::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:04:54,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:04:54,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124296 s, 330 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:04:55,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:04:55,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000926804 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:05:04,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:05:04,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772268 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:05:05,596::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:05:05,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014908 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:05:14,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:05:14,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000934333 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:05:15,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:05:15,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728083 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:05:24,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:05:24,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826022 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:05:25,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:05:25,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178805 s, 229 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:05:34,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:05:34,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144183 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:05:35,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:05:35,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157701 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:05:44,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:05:44,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.02726 s, 150 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:05:45,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:05:45,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145531 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14049::WARNING::2013-03-17 10:05:53,516::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:05:53,521::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:05:54,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:05:54,365::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863362 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:05:56,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:05:56,054::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818627 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:06:03,532::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:06:03,534::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:06:03,962::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:06:03,969::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:06:03,970::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:06:04,373::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:06:04,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017242 s, 238 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:06:06,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:06:06,140::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108446 s, 378 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:06:14,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:06:14,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868812 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:06:16,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:06:16,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140965 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:06:24,465::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:06:24,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014493 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:06:26,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:06:26,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015198 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:06:34,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:06:34,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170421 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:06:36,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:06:36,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000727224 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:06:44,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:06:44,602::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000956497 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:06:46,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:06:46,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791505 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:06:54,614::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:06:54,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807138 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:06:56,481::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:06:56,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883931 s, 4.6 MB/s\n'; <rc> = 0
>Thread-14050::WARNING::2013-03-17 10:07:03,998::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:07:04,001::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:07:04,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:07:04,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143582 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:07:06,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:07:06,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870369 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:07:14,003::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:07:14,004::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:07:14,405::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:07:14,412::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:07:14,413::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:07:14,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:07:14,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115158 s, 356 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:07:16,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:07:16,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154535 s, 265 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:07:24,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:07:24,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146956 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:07:26,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:07:26,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142553 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:07:34,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:07:34,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791194 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:07:36,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:07:36,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088542 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:07:44,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:07:44,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014835 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:07:46,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:07:46,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00733836 s, 558 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:07:54,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:07:54,907::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087126 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:07:56,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:07:57,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00182622 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:08:04,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:08:04,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144108 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:08:07,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:08:07,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000971645 s, 4.2 MB/s\n'; <rc> = 0
>Thread-14051::WARNING::2013-03-17 10:08:14,444::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:08:14,447::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:08:14,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:08:14,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104199 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:08:17,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:08:17,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843162 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:08:24,448::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 10:08:24,449::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:08:24,450::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:08:24,919::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:08:24,926::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:08:24,928::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:08:24,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:08:25,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135973 s, 301 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:08:27,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:08:27,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173219 s, 236 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:08:35,063::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 10:08:35,064::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 10:08:35,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:08:35,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000901748 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:08:35,092::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000901748 s, 2.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 10:08:35,092::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 10:08:35,093::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 10:08:35,094::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 10:08:35,095::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:08:35,096::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:08:35,096::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:08:35,097::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:08:35,097::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:08:35,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:08:35,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:08:35,542::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:08:35,543::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 10:08:35,545::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 10:08:35,546::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 10:08:35,547::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 10:08:35,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:08:35,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:08:35,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:08:35,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124199 s, 330 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:08:37,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:08:37,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133913 s, 306 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:08:45,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:08:46,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137796 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:08:47,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:08:47,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743978 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:08:56,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:08:56,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131657 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:08:57,461::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 10:08:57,463::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 10:08:57,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:08:57,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00132112 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:08:57,503::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00132112 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 10:08:57,504::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 10:08:57,504::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 10:08:57,506::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 10:08:57,507::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:08:57,507::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:08:57,508::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:08:57,508::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:08:57,509::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:08:57,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:08:57,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:08:57,921::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:08:57,922::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 10:08:57,925::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 10:08:57,926::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 10:08:57,926::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 10:08:57,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:08:58,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:08:58,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:08:58,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00249311 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:09:06,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:09:06,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756167 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:09:08,365::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:09:08,398::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00093907 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:09:16,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:09:16,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00073304 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:09:18,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:09:18,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872075 s, 4.7 MB/s\n'; <rc> = 0
>Thread-14052::WARNING::2013-03-17 10:09:24,953::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:09:24,957::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:09:26,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:09:26,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000951805 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:09:28,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:09:28,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078318 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:09:34,959::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:09:34,960::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:09:35,361::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:09:35,368::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:09:35,369::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:09:36,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:09:36,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121315 s, 338 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:09:38,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:09:38,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104683 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:09:46,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:09:46,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146214 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:09:48,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:09:48,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855695 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:09:56,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:09:56,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871724 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:09:58,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:09:58,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769895 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:10:06,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:10:06,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792686 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:10:08,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:10:08,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128309 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:10:16,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:10:16,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766546 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:10:18,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:10:18,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000981725 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:10:26,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:10:26,424::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814108 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:10:28,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:10:29,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0157276 s, 260 kB/s\n'; <rc> = 0
>Thread-14053::WARNING::2013-03-17 10:10:35,399::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:10:35,403::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:10:36,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:10:36,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815819 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:10:39,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:10:39,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916481 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:10:45,405::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:10:45,406::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:10:45,834::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:10:45,840::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:10:45,842::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:10:46,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:10:46,512::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135886 s, 301 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:10:49,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:10:49,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101266 s, 404 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:10:56,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:10:56,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000711 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:10:59,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:10:59,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00187785 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:11:06,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:11:06,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800921 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:11:09,341::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:11:09,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127575 s, 321 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:11:16,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:11:16,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146313 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:11:19,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:11:19,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079236 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:11:26,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:11:26,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000937983 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:11:29,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:11:29,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151719 s, 270 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:11:36,681::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:11:36,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148939 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:11:39,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:11:39,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843107 s, 4.9 MB/s\n'; <rc> = 0
>Thread-14054::WARNING::2013-03-17 10:11:45,872::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:11:45,876::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:11:46,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:11:46,766::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0038163 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:11:49,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:11:49,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185528 s, 221 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:11:55,878::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:11:55,880::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:11:56,285::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:11:56,291::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:11:56,293::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:11:56,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:11:56,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105622 s, 388 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:11:59,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:11:59,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01249 s, 328 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:12:06,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:12:06,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00307782 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:12:09,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:12:09,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140359 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:12:16,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:12:16,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853865 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:12:19,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:12:19,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864402 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:12:26,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:12:26,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862854 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:12:29,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:12:30,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133081 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:12:36,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:12:36,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793027 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:12:40,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:12:40,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138275 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:12:46,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:12:47,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00097106 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:12:50,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:12:50,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00654249 s, 626 kB/s\n'; <rc> = 0
>Thread-14055::WARNING::2013-03-17 10:12:56,321::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:12:56,327::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:12:57,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:12:57,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805896 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:13:00,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:13:00,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891172 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:13:06,330::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:13:06,331::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:13:06,747::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:13:06,753::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:13:06,755::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:13:07,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:13:07,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019846 s, 206 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:13:10,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:13:10,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00623399 s, 657 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:13:17,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:13:17,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084851 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:13:20,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:13:20,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141022 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:13:27,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:13:27,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140473 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:13:30,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:13:30,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135311 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:13:37,231::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 10:13:37,232::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 10:13:37,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:13:37,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000946176 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:13:37,264::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000946176 s, 2.2 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 10:13:37,264::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 10:13:37,265::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 10:13:37,266::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 10:13:37,267::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:13:37,268::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:13:37,268::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:13:37,269::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:13:37,269::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:13:37,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:13:37,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:13:37,701::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:13:37,702::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 10:13:37,705::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 10:13:37,706::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 10:13:37,706::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 10:13:37,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:13:38,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:13:38,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:13:38,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122621 s, 334 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:13:40,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:13:40,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00918668 s, 446 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:13:48,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:13:48,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016738 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:13:50,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:13:50,658::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111674 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:13:58,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:13:58,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874549 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:14:00,701::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 10:14:00,702::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 10:14:00,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:14:00,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00149698 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:14:00,742::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00149698 s, 1.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 10:14:00,742::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 10:14:00,743::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 10:14:00,744::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 10:14:00,749::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:14:00,750::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:14:00,750::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:14:00,751::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:14:00,751::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:14:00,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:14:01,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:14:01,233::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:14:01,234::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 10:14:01,249::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 10:14:01,250::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 10:14:01,251::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 10:14:01,254::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:14:01,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:14:01,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:14:01,765::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00914397 s, 448 kB/s\n'; <rc> = 0
>Thread-14056::WARNING::2013-03-17 10:14:06,784::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:14:06,786::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:14:08,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:14:08,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818149 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:14:11,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:14:11,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147786 s, 2.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:14:16,788::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 10:14:16,789::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:14:16,790::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:14:17,261::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:14:17,268::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:14:17,270::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:14:18,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:14:18,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176472 s, 232 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:14:21,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:14:21,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00971554 s, 422 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:14:28,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:14:28,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159563 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:14:31,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:14:31,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854094 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:14:38,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:14:38,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141517 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:14:42,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:14:42,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789128 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:14:48,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:14:48,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130345 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:14:52,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:14:52,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781729 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:14:58,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:14:58,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151716 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:15:02,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:15:02,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159293 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:15:08,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:15:08,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816472 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:15:12,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:15:12,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127244 s, 3.2 MB/s\n'; <rc> = 0
>Thread-14057::WARNING::2013-03-17 10:15:17,296::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:15:17,300::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:15:18,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:15:18,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117999 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:15:22,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:15:22,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0306793 s, 134 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:15:27,302::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:15:27,303::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:15:27,725::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:15:27,732::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:15:27,733::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:15:28,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:15:28,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127898 s, 320 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:15:32,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:15:32,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769531 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:15:38,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:15:38,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110323 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:15:42,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:15:42,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154363 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:15:48,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:15:48,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167146 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:15:52,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:15:52,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768626 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:15:58,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:15:58,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141466 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:16:02,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:16:02,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000993637 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:16:08,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:16:08,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00293639 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:16:12,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:16:12,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855726 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:16:18,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:16:18,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00387333 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:16:22,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:16:22,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014724 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14058::WARNING::2013-03-17 10:16:27,760::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:16:27,764::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:16:28,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:16:28,907::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795648 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:16:32,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:16:32,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.039304 s, 104 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:16:37,768::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:16:37,769::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:16:38,172::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:16:38,179::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:16:38,180::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:16:38,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:16:38,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185238 s, 221 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:16:42,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:16:43,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142579 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:16:48,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:16:49,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916621 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:16:53,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:16:53,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847508 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:16:59,014::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:16:59,040::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815414 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:17:03,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:17:03,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143902 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:17:09,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:17:09,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168822 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:17:13,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:17:13,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137005 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:17:19,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:17:19,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139046 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:17:23,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:17:23,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143646 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:17:29,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:17:29,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083021 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:17:33,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:17:33,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858558 s, 4.8 MB/s\n'; <rc> = 0
>Thread-14059::WARNING::2013-03-17 10:17:38,210::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:17:38,217::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:17:39,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:17:39,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764557 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:17:43,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:17:43,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074885 s, 5.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:17:48,220::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:17:48,221::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:17:48,651::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:17:48,657::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:17:48,659::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:17:49,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:17:49,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191748 s, 214 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:17:53,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:17:53,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850408 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:17:59,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:17:59,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137886 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:18:03,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:18:03,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894794 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:18:09,345::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:18:09,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119828 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:18:13,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:18:13,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807101 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:18:19,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:18:19,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801604 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:18:23,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:18:23,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00202836 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:18:29,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:18:29,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748453 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:18:33,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:18:33,837::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146013 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:18:39,466::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 10:18:39,467::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 10:18:39,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:18:39,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00136895 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:18:39,504::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00136895 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 10:18:39,504::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 10:18:39,504::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 10:18:39,506::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 10:18:39,507::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:18:39,508::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:18:39,508::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:18:39,508::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:18:39,509::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:18:39,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:18:39,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:18:39,936::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:18:39,937::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 10:18:39,940::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 10:18:39,941::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 10:18:39,941::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 10:18:39,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:18:40,350::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:18:40,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:18:40,399::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181246 s, 226 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:18:43,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:18:43,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00403601 s, 1.0 MB/s\n'; <rc> = 0
>Thread-14060::WARNING::2013-03-17 10:18:48,695::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:18:48,701::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:18:50,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:18:50,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139836 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:18:53,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:18:54,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0273719 s, 150 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:18:58,704::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:18:58,705::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:18:59,123::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:18:59,130::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:18:59,132::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:19:00,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:19:00,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129347 s, 317 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:19:04,077::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 10:19:04,079::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 10:19:04,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:19:04,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000826232 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:19:04,110::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000826232 s, 2.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 10:19:04,110::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 10:19:04,111::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 10:19:04,112::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 10:19:04,113::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:19:04,114::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:19:04,114::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:19:04,115::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:19:04,115::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:19:04,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:19:04,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:19:04,515::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:19:04,516::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 10:19:04,518::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 10:19:04,519::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 10:19:04,520::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 10:19:04,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:19:04,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:19:04,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:19:05,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00383133 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:19:10,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:19:10,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785471 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:19:15,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:19:15,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008894 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:19:20,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:19:20,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00226293 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:19:25,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:19:25,175::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145005 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:19:30,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:19:30,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137851 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:19:35,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:19:35,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000998026 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:19:40,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:19:40,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0066829 s, 613 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:19:45,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:19:45,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165699 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:19:50,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:19:50,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000721311 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:19:55,372::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:19:55,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799065 s, 5.1 MB/s\n'; <rc> = 0
>Thread-14061::WARNING::2013-03-17 10:19:59,161::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:19:59,168::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:20:00,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:20:00,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160906 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:20:05,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:20:05,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776461 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:20:09,170::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 10:20:09,171::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:20:09,172::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:20:09,616::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:20:09,623::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:20:09,624::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:20:10,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:20:10,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155662 s, 263 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:20:15,528::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:20:15,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778565 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:20:20,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:20:20,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132719 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:20:25,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:20:25,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177694 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:20:30,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:20:30,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144818 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:20:35,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:20:35,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120667 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:20:40,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:20:40,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808605 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:20:45,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:20:45,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158128 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:20:50,922::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:20:50,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0083993 s, 488 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:20:55,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:20:55,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000938455 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:21:00,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:21:01,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810413 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:21:05,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:21:05,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154351 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14062::WARNING::2013-03-17 10:21:09,663::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:21:09,668::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:21:11,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:21:11,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141966 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:21:16,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:21:16,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904204 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:21:19,670::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:21:19,671::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:21:20,096::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:21:20,104::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:21:20,105::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:21:21,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:21:21,123::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140562 s, 291 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:21:26,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:21:26,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868401 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:21:31,128::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:21:31,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147821 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:21:36,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:21:36,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141572 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:21:41,178::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:21:41,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818114 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:21:46,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:21:46,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000712447 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:21:51,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:21:51,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757828 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:21:56,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:21:56,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00073565 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:22:01,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:22:01,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00276162 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:22:06,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:22:06,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00245021 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:22:11,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:22:11,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769516 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:22:16,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:22:16,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175038 s, 2.3 MB/s\n'; <rc> = 0
>Thread-14063::WARNING::2013-03-17 10:22:20,134::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:22:20,136::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:22:21,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:22:21,395::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801273 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:22:26,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:22:26,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824548 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:22:30,138::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:22:30,139::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:22:30,586::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:22:30,593::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:22:30,594::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:22:31,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:22:31,456::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151294 s, 271 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:22:36,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:22:36,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859079 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:22:41,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:22:41,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814095 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:22:46,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:22:46,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160372 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:22:51,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:22:51,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161407 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:22:56,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:22:56,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167222 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:23:01,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:23:01,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138293 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:23:06,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:23:06,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00180871 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:23:11,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:23:11,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139303 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:23:16,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:23:16,958::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868786 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:23:21,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:23:21,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151224 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:23:26,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:23:27,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162042 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14064::WARNING::2013-03-17 10:23:30,623::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:23:30,627::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:23:31,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:23:31,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089379 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:23:37,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:23:37,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104272 s, 3.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:23:40,630::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:23:40,631::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:23:41,077::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:23:41,084::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:23:41,085::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:23:41,716::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 10:23:41,719::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 10:23:41,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:23:41,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00992727 s, 206 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:23:41,765::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00992727 s, 206 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 10:23:41,766::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 10:23:41,766::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 10:23:41,768::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 10:23:41,769::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:23:41,769::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:23:41,770::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:23:41,770::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:23:41,771::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:23:41,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:23:42,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:23:42,162::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:23:42,163::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 10:23:42,166::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 10:23:42,167::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 10:23:42,167::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 10:23:42,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:23:42,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:23:42,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:23:42,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000674846 s, 6.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:23:47,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:23:47,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754331 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:23:52,571::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:23:52,627::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010481 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:23:57,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:23:57,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789554 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:24:02,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:24:02,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113043 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:24:07,285::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 10:24:07,286::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 10:24:07,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:24:07,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00110946 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:24:07,322::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00110946 s, 1.8 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 10:24:07,323::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 10:24:07,323::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 10:24:07,325::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 10:24:07,326::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:24:07,326::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:24:07,327::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:24:07,327::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:24:07,328::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:24:07,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:24:07,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:24:07,840::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:24:07,841::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 10:24:07,843::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 10:24:07,844::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 10:24:07,845::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 10:24:07,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:24:08,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:24:08,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:24:08,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00845936 s, 484 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:24:12,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:24:12,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00213374 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:24:18,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:24:18,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00992555 s, 413 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:24:22,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:24:22,809::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0231911 s, 177 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:24:28,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:24:28,448::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151853 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:24:32,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:24:32,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145672 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:24:38,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:24:38,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130088 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14065::WARNING::2013-03-17 10:24:41,110::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:24:41,114::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:24:42,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:24:42,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876757 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:24:48,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:24:48,631::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749244 s, 5.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:24:51,116::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:24:51,118::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:24:51,567::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:24:51,574::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:24:51,576::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:24:52,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:24:52,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155843 s, 263 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:24:58,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:24:58,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770052 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:25:02,959::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:25:03,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139687 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:25:08,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:25:08,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792993 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:25:13,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:25:13,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758914 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:25:18,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:25:18,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776528 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:25:23,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:25:23,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000962945 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:25:28,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:25:28,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151913 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:25:33,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:25:33,123::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016223 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:25:38,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:25:39,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795639 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:25:43,128::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:25:43,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141176 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:25:49,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:25:49,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785186 s, 5.2 MB/s\n'; <rc> = 0
>Thread-14066::WARNING::2013-03-17 10:25:51,605::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:25:51,610::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:25:53,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:25:53,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790256 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:25:59,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:25:59,163::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770994 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:26:01,612::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 10:26:01,613::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:26:01,614::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:26:02,092::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:26:02,100::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:26:02,109::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:26:03,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:26:03,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172012 s, 238 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:26:09,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:26:09,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000988944 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:26:13,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:26:13,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110688 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:26:19,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:26:19,350::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158762 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:26:23,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:26:23,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141006 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:26:29,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:26:29,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775983 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:26:33,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:26:33,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00398777 s, 1.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:26:39,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:26:39,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134793 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:26:43,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:26:43,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806647 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:26:49,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:26:49,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014229 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:26:53,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:26:53,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125191 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:26:59,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:26:59,700::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923339 s, 4.4 MB/s\n'; <rc> = 0
>Thread-14067::WARNING::2013-03-17 10:27:02,139::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:27:02,144::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:27:03,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:27:03,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100843 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:27:09,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:27:09,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909967 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:27:12,147::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:27:12,148::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:27:12,623::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:27:12,630::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:27:12,631::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:27:13,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:27:13,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0268108 s, 153 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:27:19,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:27:19,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899838 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:27:23,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:27:23,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810149 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:27:29,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:27:29,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794763 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:27:33,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:27:33,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126457 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:27:39,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:27:40,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162851 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:27:43,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:27:43,805::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138241 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:27:50,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:27:50,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936312 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:27:53,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:27:53,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153266 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:28:00,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:28:00,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818909 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:28:03,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:28:03,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144445 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:28:10,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:28:10,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168492 s, 2.4 MB/s\n'; <rc> = 0
>Thread-14068::WARNING::2013-03-17 10:28:12,658::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:28:12,662::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:28:13,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:28:13,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190086 s, 215 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:28:20,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:28:20,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785038 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:28:22,664::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:28:22,665::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:28:23,095::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:28:23,103::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:28:23,104::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:28:23,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:28:24,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0251372 s, 163 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:28:30,429::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:28:30,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786056 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:28:34,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:28:34,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016293 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:28:40,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:28:40,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162173 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:28:44,119::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 10:28:44,120::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 10:28:44,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:28:44,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000817323 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:28:44,152::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000817323 s, 2.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 10:28:44,153::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 10:28:44,153::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 10:28:44,155::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 10:28:44,156::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:28:44,156::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:28:44,157::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:28:44,157::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:28:44,157::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:28:44,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:28:44,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:28:44,612::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:28:44,613::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 10:28:44,616::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 10:28:44,617::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 10:28:44,617::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 10:28:44,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:28:44,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:28:44,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:28:45,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183211 s, 224 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:28:50,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:28:50,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128242 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:28:55,048::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:28:55,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00865052 s, 473 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:29:00,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:29:00,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000691169 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:29:05,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:29:05,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141557 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:29:10,761::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 10:29:10,763::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 10:29:10,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:29:10,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0144705 s, 142 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:29:10,817::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0144705 s, 142 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 10:29:10,818::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 10:29:10,818::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 10:29:10,820::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 10:29:10,821::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:29:10,821::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:29:10,822::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:29:10,823::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:29:10,823::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:29:10,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:29:11,236::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:29:11,242::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:29:11,243::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 10:29:11,245::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 10:29:11,246::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 10:29:11,247::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 10:29:11,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:29:11,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:29:11,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:29:11,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00972685 s, 421 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:29:15,156::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:29:15,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00823386 s, 497 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:29:21,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:29:21,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861473 s, 4.8 MB/s\n'; <rc> = 0
>Thread-14069::WARNING::2013-03-17 10:29:23,136::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:29:23,140::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:29:25,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:29:25,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851103 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:29:31,997::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:29:32,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877669 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:29:33,142::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:29:33,144::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:29:33,591::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:29:33,597::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:29:33,599::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:29:35,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:29:35,297::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016772 s, 244 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:29:42,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:29:42,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896873 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:29:45,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:29:45,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841793 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:29:52,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:29:52,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149974 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:29:55,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:29:55,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143546 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:30:02,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:30:02,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154035 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:30:05,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:30:05,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829382 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:30:12,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:30:12,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133237 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:30:15,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:30:15,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141295 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:30:22,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:30:22,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780563 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:30:25,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:30:25,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008737 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:30:32,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:30:32,738::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0302394 s, 135 kB/s\n'; <rc> = 0
>Thread-14070::WARNING::2013-03-17 10:30:33,631::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:30:33,636::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:30:35,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:30:35,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151519 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:30:42,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:30:43,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896495 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:30:43,639::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:30:43,647::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:30:44,109::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:30:44,116::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:30:44,117::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:30:45,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:30:45,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192191 s, 213 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:30:53,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:30:53,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901406 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:30:55,603::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:30:55,636::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857649 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:31:03,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:31:03,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157647 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:31:05,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:31:05,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00232424 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:31:13,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:31:13,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155338 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:31:15,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:31:15,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780574 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:31:23,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:31:23,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156902 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:31:25,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:31:25,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014921 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:31:33,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:31:33,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728301 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:31:35,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:31:35,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000959373 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:31:43,606::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:31:43,671::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802378 s, 5.1 MB/s\n'; <rc> = 0
>Thread-14071::WARNING::2013-03-17 10:31:44,152::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:31:44,154::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:31:45,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:31:45,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831399 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:31:53,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:31:53,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869348 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:31:54,156::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 10:31:54,157::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:31:54,159::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:31:54,610::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:31:54,617::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:31:54,618::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:31:55,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:31:55,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131671 s, 311 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:32:03,859::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:32:03,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155873 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:32:05,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:32:05,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082291 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:32:13,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:32:13,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00263945 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:32:15,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:32:15,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923684 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:32:24,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:32:24,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117325 s, 349 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:32:25,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:32:26,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144205 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:32:34,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:32:34,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743822 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:32:36,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:32:36,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013329 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:32:44,171::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:32:44,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811672 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:32:46,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:32:46,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135846 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:32:54,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:32:54,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00259835 s, 1.6 MB/s\n'; <rc> = 0
>Thread-14072::WARNING::2013-03-17 10:32:54,663::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:32:54,667::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:32:56,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:32:56,140::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759921 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:33:04,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:33:04,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758039 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:33:04,694::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:33:04,695::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:33:05,173::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:33:05,181::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:33:05,183::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:33:06,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:33:06,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190303 s, 215 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:33:14,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:33:14,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129804 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:33:16,209::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:33:16,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00196952 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:33:24,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:33:24,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855529 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:33:26,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:33:26,297::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164061 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:33:34,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:33:34,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015339 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:33:36,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:33:36,331::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809159 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:33:44,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:33:44,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803955 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:33:46,336::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 10:33:46,337::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 10:33:46,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:33:46,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00073007 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:33:46,372::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00073007 s, 2.8 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 10:33:46,372::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 10:33:46,373::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 10:33:46,374::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 10:33:46,375::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:33:46,376::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:33:46,376::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:33:46,377::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:33:46,377::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:33:46,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:33:46,809::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:33:46,814::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:33:46,815::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 10:33:46,817::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 10:33:46,818::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 10:33:46,819::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 10:33:46,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:33:47,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:33:47,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:33:47,236::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141121 s, 290 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:33:54,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:33:55,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154156 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:33:57,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:33:57,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014256 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:34:05,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:34:05,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875213 s, 4.7 MB/s\n'; <rc> = 0
>Thread-14073::WARNING::2013-03-17 10:34:05,214::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:34:05,226::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:34:07,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:34:07,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859197 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:34:15,232::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:34:15,234::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:34:15,259::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 10:34:15,262::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 10:34:15,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:34:15,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0136641 s, 150 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:34:15,321::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0136641 s, 150 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 10:34:15,322::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 10:34:15,322::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 10:34:15,324::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 10:34:15,325::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm reload operation' is holding the operation mutex, waiting...
>Thread-21::DEBUG::2013-03-17 10:34:15,719::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:34:15,727::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:34:15,727::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:34:15,732::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:34:15,733::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:34:15,733::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:34:15,734::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:34:15,734::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:34:15,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:34:16,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:34:16,148::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:34:16,149::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 10:34:16,152::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 10:34:16,153::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 10:34:16,153::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 10:34:16,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:34:16,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:34:16,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:34:16,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112395 s, 364 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:34:17,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:34:17,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100632 s, 407 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:34:26,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:34:26,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815711 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:34:27,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:34:27,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104258 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:34:36,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:34:36,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872773 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:34:37,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:34:37,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810435 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:34:46,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:34:46,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000670252 s, 6.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:34:47,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:34:47,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146136 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:34:56,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:34:56,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791951 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:34:57,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:34:57,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144581 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:35:06,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:35:06,890::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793931 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:35:07,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:35:07,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140938 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14074::WARNING::2013-03-17 10:35:15,814::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:35:15,817::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 10:35:16,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:35:16,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790843 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:35:17,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:35:17,622::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826923 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:35:25,819::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:35:25,820::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:35:26,225::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:35:26,231::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:35:26,233::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:35:26,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:35:27,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00882487 s, 464 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:35:27,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:35:27,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115276 s, 355 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:35:37,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:35:37,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081478 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:35:37,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:35:37,712::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156246 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:35:47,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:35:47,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831281 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:35:47,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:35:47,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100412 s, 408 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:35:57,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:35:57,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000736884 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:35:57,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:35:57,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000942955 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:36:07,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:36:07,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0306068 s, 134 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:36:07,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:36:07,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014095 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:36:17,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:36:17,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832216 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:36:17,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:36:17,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00307013 s, 1.3 MB/s\n'; <rc> = 0
>Thread-14075::WARNING::2013-03-17 10:36:26,258::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:36:26,262::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 10:36:27,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:36:27,511::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154253 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:36:27,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:36:27,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852912 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:36:36,277::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:36:36,278::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:36:36,688::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:36:36,695::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:36:36,696::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:36:37,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:36:37,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138056 s, 297 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:36:37,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:36:38,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188072 s, 218 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:36:47,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:36:47,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810887 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:36:48,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:36:48,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00213057 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:36:57,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:36:57,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145318 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:36:58,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:36:58,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176872 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:37:07,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:37:07,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015241 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:37:08,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:37:08,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797797 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:37:17,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:37:17,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152541 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:37:18,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:37:18,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105303 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:37:27,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:37:27,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866153 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:37:28,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:37:28,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120461 s, 3.4 MB/s\n'; <rc> = 0
>Thread-14076::WARNING::2013-03-17 10:37:36,722::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:37:36,726::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 10:37:38,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:37:38,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812577 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:37:38,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:37:38,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789458 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:37:46,728::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 10:37:46,729::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:37:46,729::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:37:47,203::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:37:47,209::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:37:47,211::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:37:48,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:37:48,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0254945 s, 161 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:37:48,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:37:48,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100493 s, 408 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:37:58,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:37:58,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000957537 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:37:58,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:37:58,511::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107786 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:38:08,280::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:38:08,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762233 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:38:08,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:38:08,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017651 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:38:18,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:38:18,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00065709 s, 6.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:38:18,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:38:18,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136963 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:38:28,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:38:28,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145068 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:38:28,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:38:28,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00239799 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:38:38,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:38:38,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858519 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:38:38,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:38:38,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121343 s, 3.4 MB/s\n'; <rc> = 0
>Thread-14077::WARNING::2013-03-17 10:38:47,244::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:38:47,250::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 10:38:48,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:38:48,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00590543 s, 694 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:38:48,774::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 10:38:48,776::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 10:38:48,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:38:48,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00204491 s, 1.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:38:48,842::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00204491 s, 1.0 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 10:38:48,843::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 10:38:48,843::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 10:38:48,844::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 10:38:48,845::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:38:48,846::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:38:48,847::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:38:48,847::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:38:48,847::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:38:48,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:38:49,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:38:49,322::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:38:49,323::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 10:38:49,325::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 10:38:49,326::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 10:38:49,327::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 10:38:49,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:38:49,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:38:49,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:38:49,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173621 s, 236 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:38:57,258::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:38:57,259::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:38:57,740::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:38:57,747::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:38:57,749::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:38:58,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:38:58,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111892 s, 366 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:38:59,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:38:59,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136274 s, 301 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:39:08,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:39:08,765::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100312 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:39:09,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:39:09,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840678 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:39:18,806::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 10:39:18,807::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 10:39:18,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:39:18,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000729802 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:39:18,838::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000729802 s, 2.8 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 10:39:18,838::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 10:39:18,839::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 10:39:18,840::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 10:39:18,841::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:39:18,842::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:39:18,843::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:39:18,843::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:39:18,843::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:39:18,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:39:19,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:39:19,295::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:39:19,296::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 10:39:19,298::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 10:39:19,300::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 10:39:19,300::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 10:39:19,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:39:19,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:39:19,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:39:19,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00614015 s, 667 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:39:19,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:39:19,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168419 s, 243 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:39:29,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:39:29,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00236284 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:39:29,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:39:29,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157995 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:39:39,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:39:39,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151446 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:39:39,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:39:39,979::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000730179 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:39:49,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:39:49,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899257 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:39:49,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:39:50,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844603 s, 4.8 MB/s\n'; <rc> = 0
>Thread-14078::WARNING::2013-03-17 10:39:57,774::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:39:57,778::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 10:39:59,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:40:00,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086377 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:40:00,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:40:00,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00093911 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:40:07,781::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:40:07,782::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:40:08,218::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:40:08,225::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:40:08,226::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:40:10,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:40:10,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:40:10,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104355 s, 393 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:40:10,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146367 s, 280 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:40:20,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:40:20,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737527 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:40:20,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:40:20,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904831 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:40:30,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:40:30,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000921539 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:40:30,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:40:30,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000935002 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:40:40,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:40:40,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832344 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:40:40,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:40:40,456::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000953732 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:40:50,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:40:50,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.029034 s, 141 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:40:50,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:40:50,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103924 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:41:00,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:41:00,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145397 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:41:00,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:41:00,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165316 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14079::WARNING::2013-03-17 10:41:08,254::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:41:08,258::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:41:10,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:41:10,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822314 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:41:10,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:41:10,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000935291 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:41:18,261::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:41:18,262::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:41:18,692::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:41:18,699::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:41:18,700::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:41:20,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:41:20,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174337 s, 235 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:41:20,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:41:20,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807739 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:41:30,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:41:30,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158228 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:41:30,782::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:41:30,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157087 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:41:40,603::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:41:40,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134721 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:41:40,861::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:41:40,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000918727 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:41:50,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:41:50,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00850716 s, 481 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:41:50,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:41:50,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802484 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:42:00,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:42:00,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131031 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:42:01,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:42:01,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807385 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:42:10,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:42:10,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153239 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:42:11,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:42:11,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014008 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14080::WARNING::2013-03-17 10:42:18,740::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:42:18,743::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:42:20,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:42:20,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00231502 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:42:21,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:42:21,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085206 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:42:28,745::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:42:28,746::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:42:29,157::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:42:29,164::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:42:29,165::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:42:30,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:42:30,912::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168195 s, 244 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:42:31,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:42:31,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00096809 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:42:40,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:42:40,950::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814574 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:42:41,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:42:41,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163496 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:42:50,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:42:50,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858806 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:42:51,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:42:51,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834581 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:43:00,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:43:01,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125373 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:43:01,439::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:43:01,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154898 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:43:11,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:43:11,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164098 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:43:11,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:43:11,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749896 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:43:21,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:43:21,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156414 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:43:21,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:43:21,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016154 s, 254 kB/s\n'; <rc> = 0
>Thread-14081::WARNING::2013-03-17 10:43:29,192::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:43:29,195::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:43:31,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:43:31,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864276 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:43:31,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:43:31,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000974001 s, 4.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:43:39,197::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 10:43:39,198::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:43:39,199::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:43:39,624::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:43:39,630::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:43:39,632::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:43:41,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:43:41,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112341 s, 365 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:43:41,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:43:41,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854619 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:43:51,201::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 10:43:51,203::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 10:43:51,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:43:51,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000725049 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:43:51,231::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000725049 s, 2.8 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 10:43:51,231::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 10:43:51,231::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 10:43:51,233::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 10:43:51,234::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:43:51,235::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:43:51,235::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:43:51,236::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:43:51,236::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:43:51,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:43:51,649::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:43:51,654::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:43:51,656::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 10:43:51,659::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 10:43:51,660::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 10:43:51,661::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 10:43:51,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:43:51,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:43:51,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014896 s, 275 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:43:52,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:43:52,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:43:52,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182609 s, 224 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:44:01,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:44:01,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123094 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:44:02,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:44:02,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114382 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:44:12,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:44:12,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172203 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:44:12,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:44:12,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00302227 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:44:22,117::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 10:44:22,118::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 10:44:22,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:44:22,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00145146 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:44:22,162::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00145146 s, 1.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 10:44:22,162::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 10:44:22,163::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 10:44:22,164::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 10:44:22,165::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:44:22,166::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:44:22,166::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:44:22,167::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:44:22,167::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:44:22,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:44:22,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:44:22,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153787 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:44:22,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:44:22,609::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:44:22,610::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 10:44:22,612::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 10:44:22,613::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 10:44:22,614::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 10:44:22,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:44:22,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:44:22,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:44:23,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00713873 s, 574 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:44:32,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:44:32,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140027 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:44:33,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:44:33,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0006771 s, 6.0 MB/s\n'; <rc> = 0
>Thread-14082::WARNING::2013-03-17 10:44:39,664::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:44:39,668::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:44:42,341::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:44:42,366::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805789 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:44:43,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:44:43,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105267 s, 3.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:44:49,671::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:44:49,672::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:44:50,076::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:44:50,083::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:44:50,084::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:44:52,372::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:44:52,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150587 s, 272 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:44:53,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:44:53,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082007 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:45:02,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:45:02,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824523 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:45:03,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:45:03,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781177 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:45:12,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:45:12,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161018 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:45:13,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:45:13,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830928 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:45:22,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:45:22,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149949 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:45:23,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:45:23,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140533 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:45:32,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:45:32,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140788 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:45:33,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:45:33,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156786 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:45:42,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:45:42,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162367 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:45:43,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:45:43,631::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0068279 s, 600 kB/s\n'; <rc> = 0
>Thread-14083::WARNING::2013-03-17 10:45:50,109::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:45:50,113::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:45:52,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:45:52,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144385 s, 284 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:45:53,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:45:53,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788082 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:46:00,121::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:46:00,123::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:46:00,575::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:46:00,583::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:46:00,585::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:46:02,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:46:02,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111032 s, 369 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:46:03,741::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:46:03,766::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851323 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:46:12,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:46:12,793::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197314 s, 208 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:46:13,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:46:13,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124147 s, 330 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:46:22,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:46:22,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140688 s, 291 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:46:23,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:46:23,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152019 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:46:32,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:46:32,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806939 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:46:33,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:46:34,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143894 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:46:42,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:46:42,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799504 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:46:44,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:46:44,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00923094 s, 444 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:46:52,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:46:52,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156232 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:46:54,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:46:54,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157783 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14084::WARNING::2013-03-17 10:47:00,620::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:47:00,623::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:47:02,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:47:03,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114469 s, 358 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:47:04,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:47:04,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106949 s, 3.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:47:10,632::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:47:10,633::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:47:11,063::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:47:11,070::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:47:11,071::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:47:13,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:47:13,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146395 s, 280 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:47:14,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:47:14,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814553 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:47:23,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:47:23,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0255559 s, 160 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:47:24,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:47:24,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017073 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:47:33,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:47:33,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000682297 s, 6.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:47:34,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:47:34,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000930291 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:47:43,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:47:43,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768829 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:47:44,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:47:44,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151221 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:47:53,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:47:53,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774318 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:47:54,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:47:54,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144027 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:48:03,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:48:03,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173269 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:48:04,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:48:04,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144138 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14085::WARNING::2013-03-17 10:48:11,100::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:48:11,104::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:48:13,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:48:13,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763078 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:48:14,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:48:14,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100319 s, 4.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:48:21,111::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:48:21,112::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:48:21,516::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:48:21,523::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:48:21,524::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:48:23,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:48:23,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183553 s, 223 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:48:24,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:48:24,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794163 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:48:33,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:48:33,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145099 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:48:34,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:48:34,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798754 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:48:43,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:48:43,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147405 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:48:44,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:48:44,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110258 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:48:53,529::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 10:48:53,530::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 10:48:53,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:48:53,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00146031 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:48:53,607::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00146031 s, 1.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 10:48:53,607::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 10:48:53,608::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 10:48:53,609::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 10:48:53,610::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:48:53,611::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:48:53,611::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:48:53,612::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:48:53,612::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:48:53,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:48:54,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:48:54,051::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:48:54,052::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 10:48:54,054::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 10:48:54,055::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 10:48:54,056::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 10:48:54,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:48:54,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:48:54,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:48:54,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151707 s, 270 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:48:55,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:48:55,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149883 s, 273 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:49:04,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:49:04,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104169 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:49:05,118::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:49:05,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085235 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:49:14,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:49:14,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873619 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:49:15,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:49:15,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764798 s, 5.4 MB/s\n'; <rc> = 0
>Thread-14086::WARNING::2013-03-17 10:49:21,555::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:49:21,561::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:49:24,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:49:24,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000976643 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:49:25,266::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 10:49:25,268::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 10:49:25,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:49:25,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000758553 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:49:25,295::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000758553 s, 2.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 10:49:25,296::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 10:49:25,296::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 10:49:25,298::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 10:49:25,299::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:49:25,300::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:49:25,300::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:49:25,300::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:49:25,301::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:49:25,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:49:25,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:49:25,736::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:49:25,738::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 10:49:25,740::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 10:49:25,741::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 10:49:25,742::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 10:49:25,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:49:26,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:49:26,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:49:26,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00538628 s, 760 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:49:31,563::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 10:49:31,564::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:49:31,565::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:49:31,977::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:49:31,984::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:49:31,986::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:49:34,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:49:34,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147697 s, 277 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:49:36,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:49:36,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167026 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:49:44,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:49:44,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771631 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:49:46,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:49:46,270::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834454 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:49:54,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:49:54,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000943117 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:49:56,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:49:56,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861973 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:50:04,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:50:04,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013542 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:50:06,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:50:06,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151543 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:50:14,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:50:14,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140222 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:50:16,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:50:16,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847288 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:50:24,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:50:24,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156527 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:50:26,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:50:26,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00314705 s, 1.3 MB/s\n'; <rc> = 0
>Thread-14087::WARNING::2013-03-17 10:50:32,013::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:50:32,017::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:50:34,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:50:34,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149548 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:50:36,625::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:50:36,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000957362 s, 4.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:50:42,019::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:50:42,020::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:50:42,515::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:50:42,521::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:50:42,523::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:50:44,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:50:44,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010329 s, 397 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:50:46,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:50:46,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800648 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:50:54,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:50:55,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0224413 s, 183 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:50:56,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:50:56,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086747 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:51:05,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:51:05,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137124 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:51:06,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:51:06,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169607 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:51:15,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:51:15,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000735775 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:51:16,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:51:16,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140699 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:51:25,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:51:25,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00284543 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:51:26,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:51:27,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169275 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:51:35,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:51:35,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134208 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:51:37,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:51:37,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150708 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14088::WARNING::2013-03-17 10:51:42,556::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:51:42,562::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:51:45,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:51:45,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000939393 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:51:47,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:51:47,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000946865 s, 4.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:51:52,564::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:51:52,565::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:51:53,022::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:51:53,029::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:51:53,030::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:51:55,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:51:55,311::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123955 s, 330 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:51:57,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:51:57,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137538 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:52:05,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:52:05,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898638 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:52:07,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:52:07,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136717 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:52:15,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:52:15,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000696779 s, 5.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:52:17,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:52:17,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906959 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:52:25,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:52:25,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147835 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:52:27,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:52:27,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107161 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:52:35,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:52:35,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142013 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:52:37,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:52:37,562::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139107 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:52:45,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:52:45,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152544 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:52:47,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:52:47,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763156 s, 5.4 MB/s\n'; <rc> = 0
>Thread-14089::WARNING::2013-03-17 10:52:53,054::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:52:53,058::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:52:55,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:52:55,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00320968 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:52:57,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:52:57,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079674 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:53:03,059::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:53:03,060::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:53:03,482::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:53:03,489::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:53:03,490::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:53:05,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:53:05,591::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165967 s, 247 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:53:07,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:53:07,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870572 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:53:15,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:53:15,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088725 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:53:17,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:53:17,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000991768 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:53:25,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:53:25,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000996609 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:53:27,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:53:27,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000935706 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:53:35,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:53:35,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128484 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:53:37,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:53:37,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822569 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:53:45,717::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:53:45,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154696 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:53:48,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:53:48,046::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858361 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:53:55,763::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 10:53:55,767::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 10:53:55,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:53:55,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00802862 s, 255 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:53:55,808::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00802862 s, 255 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 10:53:55,808::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 10:53:55,809::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 10:53:55,810::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 10:53:55,811::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:53:55,812::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:53:55,812::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:53:55,812::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:53:55,813::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:53:55,814::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:53:56,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:53:56,241::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:53:56,243::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 10:53:56,245::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 10:53:56,246::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 10:53:56,247::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 10:53:56,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:53:56,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:53:56,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:53:56,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137853 s, 297 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:53:58,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:53:58,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165972 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14090::WARNING::2013-03-17 10:54:03,539::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:54:03,548::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:54:06,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:54:06,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199142 s, 206 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:54:08,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:54:08,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797116 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:54:13,550::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:54:13,551::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:54:14,064::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:54:14,071::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:54:14,072::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:54:16,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:54:16,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111721 s, 367 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:54:18,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:54:18,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798117 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:54:26,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:54:26,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153877 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:54:28,301::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 10:54:28,302::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 10:54:28,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:54:28,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000818898 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:54:28,331::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000818898 s, 2.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 10:54:28,331::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 10:54:28,332::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 10:54:28,334::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 10:54:28,335::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:54:28,335::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:54:28,336::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:54:28,336::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:54:28,336::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:54:28,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:54:28,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:54:28,751::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:54:28,753::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 10:54:28,755::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 10:54:28,756::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 10:54:28,757::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 10:54:28,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:54:29,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:54:29,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:54:29,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130121 s, 315 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:54:36,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:54:36,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000937057 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:54:39,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:54:39,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936805 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:54:46,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:54:46,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813649 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:54:49,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:54:49,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881607 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:54:56,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:54:56,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000987418 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:54:59,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:54:59,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079092 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:55:06,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:55:06,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197957 s, 207 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:55:09,396::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:55:09,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135573 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14091::WARNING::2013-03-17 10:55:14,096::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:55:14,100::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:55:16,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:55:17,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777297 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:55:19,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:55:19,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854651 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:55:24,101::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 10:55:24,102::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:55:24,103::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:55:24,501::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:55:24,507::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:55:24,509::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:55:27,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:55:27,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101184 s, 405 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:55:29,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:55:29,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116605 s, 351 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:55:37,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:55:37,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084364 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:55:39,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:55:39,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014356 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:55:47,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:55:47,140::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150208 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:55:49,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:55:49,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150302 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:55:57,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:55:57,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859322 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:55:59,792::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:55:59,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775448 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:56:07,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:56:07,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161637 s, 253 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:56:09,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:56:09,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165913 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:56:17,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:56:17,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127479 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:56:19,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:56:19,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157129 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14092::WARNING::2013-03-17 10:56:24,538::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:56:24,541::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:56:27,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:56:27,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000973847 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:56:30,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:56:30,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920834 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:56:34,543::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:56:34,544::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:56:34,942::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:56:34,948::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:56:34,950::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:56:37,324::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:56:37,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112076 s, 365 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:56:40,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:56:40,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844956 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:56:47,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:56:47,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860278 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:56:50,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:56:50,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840183 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:56:57,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:56:57,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841483 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:57:00,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:57:00,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819381 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:57:07,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:57:07,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150194 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:57:10,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:57:10,365::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000930332 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:57:17,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:57:17,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015521 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:57:20,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:57:20,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000745329 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:57:27,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:57:27,552::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786573 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:57:30,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:57:30,513::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170948 s, 2.4 MB/s\n'; <rc> = 0
>Thread-14093::WARNING::2013-03-17 10:57:34,981::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:57:34,984::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:57:37,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:57:37,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843602 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:57:40,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:57:40,593::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848712 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:57:44,986::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:57:44,988::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:57:45,420::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:57:45,426::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:57:45,428::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:57:47,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:57:47,636::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160023 s, 256 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:57:50,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:57:50,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113692 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:57:57,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:57:57,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824066 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:58:00,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:58:00,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000720594 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:58:07,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:58:07,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000705527 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:58:10,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:58:10,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000961607 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:58:17,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:58:17,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829946 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:58:20,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:58:20,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00202612 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:58:27,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:58:27,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00241444 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:58:30,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:58:30,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147567 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:58:37,796::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:58:37,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00183795 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:58:40,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:58:41,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147403 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14094::WARNING::2013-03-17 10:58:45,454::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:58:45,457::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:58:47,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:58:47,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104222 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:58:51,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:58:51,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812367 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 10:58:55,459::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 10:58:55,460::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 10:58:55,913::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 10:58:55,920::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 10:58:55,922::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:58:57,878::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 10:58:57,881::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 10:58:57,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:58:57,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0152509 s, 134 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:58:57,936::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0152509 s, 134 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 10:58:57,937::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 10:58:57,937::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 10:58:57,939::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 10:58:57,940::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:58:57,940::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:58:57,941::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:58:57,941::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:58:57,941::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 10:58:57,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:58:58,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:58:58,340::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 10:58:58,342::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 10:58:58,344::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 10:58:58,345::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 10:58:58,346::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 10:58:58,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:58:58,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:58:58,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:58:58,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178202 s, 230 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:59:01,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:59:01,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078948 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:59:08,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:59:08,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879694 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:59:11,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:59:11,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00263856 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:59:18,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:59:18,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082093 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:59:21,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:59:21,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016895 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:59:28,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:59:28,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000908543 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:59:31,374::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 10:59:31,376::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 10:59:31,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:59:31,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00152639 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:59:31,422::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00152639 s, 1.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 10:59:31,423::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 10:59:31,423::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 10:59:31,424::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 10:59:31,425::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:59:31,426::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:59:31,426::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:59:31,427::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:59:31,427::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 10:59:31,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:59:31,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:59:31,859::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 10:59:31,860::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 10:59:31,863::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 10:59:31,864::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 10:59:31,865::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 10:59:31,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:59:32,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:59:32,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:59:32,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00368849 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:59:38,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:59:38,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861252 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:59:42,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:59:42,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153438 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 10:59:48,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:59:48,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817581 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 10:59:52,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 10:59:52,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817375 s, 5.0 MB/s\n'; <rc> = 0
>Thread-14095::WARNING::2013-03-17 10:59:55,948::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 10:59:55,955::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 10:59:58,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 10:59:58,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808092 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:00:02,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:00:02,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866684 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:00:05,957::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:00:05,958::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:00:06,409::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:00:06,416::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:00:06,417::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:00:08,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:00:09,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140958 s, 291 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:00:12,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:00:12,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000927731 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:00:19,034::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:00:19,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100819 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:00:22,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:00:22,622::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835382 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:00:29,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:00:29,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154552 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:00:32,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:00:32,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775139 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:00:39,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:00:39,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760707 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:00:42,730::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:00:42,765::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170364 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:00:49,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:00:49,195::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170832 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:00:52,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:00:52,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144953 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:00:59,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:00:59,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156015 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:01:02,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:01:02,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139393 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14096::WARNING::2013-03-17 11:01:06,445::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:01:06,449::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:01:09,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:01:09,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140577 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:01:12,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:01:13,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936887 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:01:16,450::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 11:01:16,451::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:01:16,452::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:01:16,931::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:01:16,940::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:01:16,942::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:01:19,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:01:19,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154429 s, 265 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:01:23,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:01:23,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087855 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:01:29,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:01:29,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134245 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:01:33,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:01:33,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821995 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:01:39,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:01:39,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161011 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:01:43,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:01:43,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141355 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:01:49,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:01:49,483::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145828 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:01:53,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:01:53,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000986067 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:01:59,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:01:59,517::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837774 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:02:03,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:02:03,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000933318 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:02:09,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:02:09,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185262 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:02:13,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:02:13,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744886 s, 5.5 MB/s\n'; <rc> = 0
>Thread-14097::WARNING::2013-03-17 11:02:16,975::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:02:16,980::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:02:19,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:02:19,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00253533 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:02:23,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:02:23,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845521 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:02:26,982::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:02:26,983::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:02:27,417::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:02:27,424::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:02:27,425::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:02:29,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:02:29,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104048 s, 394 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:02:33,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:02:33,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779621 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:02:39,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:02:39,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000921754 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:02:43,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:02:43,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924581 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:02:49,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:02:49,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162975 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:02:53,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:02:53,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888802 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:02:59,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:02:59,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887844 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:03:03,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:03:03,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141811 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:03:09,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:03:09,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00672428 s, 609 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:03:13,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:03:13,928::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149722 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:03:19,814::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:03:19,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168401 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:03:23,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:03:24,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139371 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14098::WARNING::2013-03-17 11:03:27,451::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:03:27,454::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:03:29,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:03:29,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904594 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:03:34,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:03:34,081::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822121 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:03:37,456::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:03:37,457::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:03:37,910::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:03:37,916::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:03:37,918::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:03:39,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:03:39,936::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154259 s, 266 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:03:44,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:03:44,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888295 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:03:49,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:03:49,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086939 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:03:54,194::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:03:54,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798443 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:03:59,981::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 11:03:59,982::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 11:03:59,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:04:00,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00151783 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:04:00,024::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00151783 s, 1.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 11:04:00,024::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 11:04:00,025::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 11:04:00,026::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 11:04:00,027::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:04:00,028::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:04:00,028::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:04:00,029::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:04:00,029::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:04:00,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:04:00,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:04:00,474::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:04:00,475::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 11:04:00,477::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 11:04:00,479::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 11:04:00,479::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 11:04:00,481::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:04:00,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:04:00,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:04:00,973::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0232679 s, 176 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:04:04,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:04:04,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850152 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:04:10,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:04:11,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00213752 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:04:14,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:04:14,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147497 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:04:21,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:04:21,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140936 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:04:24,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:04:24,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155862 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:04:31,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:04:31,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016752 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:04:34,497::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 11:04:34,498::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 11:04:34,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:04:34,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000809069 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:04:34,530::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000809069 s, 2.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 11:04:34,530::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 11:04:34,531::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 11:04:34,533::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 11:04:34,534::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:04:34,535::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:04:34,536::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:04:34,537::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:04:34,537::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:04:34,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:04:34,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:04:34,979::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:04:34,980::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 11:04:34,983::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 11:04:34,984::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 11:04:34,984::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 11:04:34,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:04:35,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:04:35,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:04:35,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00606909 s, 675 kB/s\n'; <rc> = 0
>Thread-14099::WARNING::2013-03-17 11:04:37,943::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:04:37,947::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:04:41,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:04:41,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00299682 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:04:45,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:04:45,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893029 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:04:47,949::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:04:47,950::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:04:48,391::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:04:48,399::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:04:48,402::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:04:51,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:04:51,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125258 s, 327 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:04:55,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:04:55,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000921848 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:05:01,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:05:01,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0367641 s, 111 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:05:05,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:05:05,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176429 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:05:11,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:05:11,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00773652 s, 529 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:05:15,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:05:15,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00270436 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:05:21,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:05:21,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924685 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:05:25,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:05:25,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765304 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:05:31,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:05:31,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131555 s, 311 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:05:35,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:05:35,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015063 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:05:41,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:05:41,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169142 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:05:45,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:05:45,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000986925 s, 4.2 MB/s\n'; <rc> = 0
>Thread-14100::WARNING::2013-03-17 11:05:48,428::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:05:48,431::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:05:51,439::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:05:51,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171999 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:05:56,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:05:56,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879188 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:05:58,434::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:05:58,435::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:05:58,860::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:05:58,867::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:05:58,868::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:06:01,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:06:01,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119328 s, 343 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:06:06,108::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:06:06,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896232 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:06:11,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:06:11,598::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088265 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:06:16,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:06:16,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785769 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:06:21,603::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:06:21,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100743 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:06:26,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:06:26,270::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865625 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:06:31,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:06:31,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135618 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:06:36,312::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:06:36,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147471 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:06:41,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:06:41,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812242 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:06:46,395::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:06:46,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157452 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:06:51,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:06:51,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141061 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:06:56,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:06:56,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00231924 s, 1.8 MB/s\n'; <rc> = 0
>Thread-14101::WARNING::2013-03-17 11:06:58,896::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:06:58,899::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:07:01,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:07:01,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082851 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:07:06,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:07:06,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764925 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:07:08,904::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 11:07:08,905::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:07:08,906::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:07:09,352::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:07:09,359::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:07:09,361::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:07:11,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:07:11,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128024 s, 320 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:07:16,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:07:16,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152289 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:07:21,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:07:21,890::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737522 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:07:26,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:07:26,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892096 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:07:31,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:07:31,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000597711 s, 6.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:07:36,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:07:36,805::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131058 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:07:41,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:07:41,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0195479 s, 210 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:07:46,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:07:46,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100457 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:07:52,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:07:52,040::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737288 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:07:56,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:07:56,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077489 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:08:02,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:08:02,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00491601 s, 833 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:08:07,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:08:07,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145667 s, 281 kB/s\n'; <rc> = 0
>Thread-14102::WARNING::2013-03-17 11:08:09,394::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:08:09,400::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:08:12,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:08:12,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879388 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:08:17,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:08:17,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087487 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:08:19,404::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:08:19,405::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:08:19,867::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:08:19,874::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:08:19,875::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:08:22,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:08:22,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164032 s, 250 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:08:27,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:08:27,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117083 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:08:32,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:08:32,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115648 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:08:37,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:08:37,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000963077 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:08:42,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:08:42,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826217 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:08:47,455::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:08:47,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185131 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:08:52,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:08:52,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144527 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:08:57,522::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:08:57,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134807 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:09:02,309::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 11:09:02,310::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 11:09:02,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:09:02,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00128842 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:09:02,349::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00128842 s, 1.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 11:09:02,350::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 11:09:02,350::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 11:09:02,353::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 11:09:02,354::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:09:02,354::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:09:02,355::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:09:02,355::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:09:02,355::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:09:02,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:09:02,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:09:02,802::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:09:02,803::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 11:09:02,806::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 11:09:02,807::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 11:09:02,807::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 11:09:02,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:09:03,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:09:03,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:09:03,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140982 s, 291 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:09:07,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:09:07,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150646 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:09:13,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:09:13,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801333 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:09:17,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:09:17,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138617 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14103::WARNING::2013-03-17 11:09:19,905::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:09:19,909::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:09:23,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:09:23,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817833 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:09:27,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:09:27,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000931027 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:09:29,911::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:09:29,912::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:09:30,377::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:09:30,384::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:09:30,386::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:09:33,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:09:33,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147843 s, 277 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:09:37,883::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 11:09:37,885::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 11:09:37,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:09:37,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000820267 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:09:37,913::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000820267 s, 2.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 11:09:37,913::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 11:09:37,914::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 11:09:37,915::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 11:09:37,916::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:09:37,917::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:09:37,917::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:09:37,918::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:09:37,918::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:09:37,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:09:38,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:09:38,349::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:09:38,350::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 11:09:38,353::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 11:09:38,354::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 11:09:38,355::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 11:09:38,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:09:38,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:09:38,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:09:38,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00679971 s, 602 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:09:43,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:09:43,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175303 s, 234 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:09:48,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:09:48,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0021793 s, 1.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:09:53,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:09:53,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151724 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:09:58,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:09:58,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00942625 s, 435 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:10:03,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:10:03,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878495 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:10:08,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:10:09,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000718011 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:10:13,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:10:13,529::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829329 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:10:19,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:10:19,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877328 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:10:23,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:10:23,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805968 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:10:29,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:10:29,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00539254 s, 760 kB/s\n'; <rc> = 0
>Thread-14104::WARNING::2013-03-17 11:10:30,424::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:10:30,429::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:10:33,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:10:33,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000996499 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:10:39,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:10:39,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161989 s, 2.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:10:40,433::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:10:40,435::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:10:40,881::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:10:40,888::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:10:40,889::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:10:43,612::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:10:43,657::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156989 s, 261 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:10:49,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:10:49,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772167 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:10:53,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:10:53,712::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130148 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:10:59,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:10:59,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141307 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:11:03,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:11:03,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795739 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:11:09,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:11:09,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00070935 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:11:13,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:11:13,819::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0207519 s, 197 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:11:19,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:11:19,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126942 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:11:23,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:11:23,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821388 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:11:29,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:11:29,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144902 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:11:33,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:11:33,899::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163442 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:11:39,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:11:40,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122891 s, 3.3 MB/s\n'; <rc> = 0
>Thread-14105::WARNING::2013-03-17 11:11:40,918::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:11:40,928::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:11:43,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:11:43,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117228 s, 349 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:11:50,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:11:50,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766981 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:11:50,932::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:11:50,934::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:11:51,383::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:11:51,389::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:11:51,390::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:11:53,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:11:53,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132679 s, 309 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:12:00,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:12:00,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813184 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:12:04,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:12:04,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0196767 s, 208 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:12:10,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:12:10,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146668 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:12:14,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:12:14,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793182 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:12:20,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:12:20,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858781 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:12:24,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:12:24,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158515 s, 258 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:12:30,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:12:30,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158558 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:12:34,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:12:34,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160785 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:12:40,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:12:40,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00265393 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:12:44,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:12:44,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129118 s, 317 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:12:50,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:12:50,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737736 s, 5.6 MB/s\n'; <rc> = 0
>Thread-14106::WARNING::2013-03-17 11:12:51,415::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:12:51,419::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:12:54,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:12:54,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864705 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:13:01,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:13:01,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161757 s, 2.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:13:01,424::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 11:13:01,426::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:13:01,427::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:13:01,918::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:13:01,924::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:13:01,926::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:13:04,280::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:13:04,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119569 s, 343 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:13:11,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:13:11,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138004 s, 297 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:13:14,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:13:14,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000735921 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:13:21,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:13:21,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152253 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:13:24,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:13:24,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121773 s, 336 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:13:31,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:13:31,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873388 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:13:34,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:13:34,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796521 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:13:41,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:13:41,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846912 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:13:44,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:13:44,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.012956 s, 316 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:13:51,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:13:51,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884749 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:13:54,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:13:54,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146892 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:14:01,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:14:01,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00705599 s, 580 kB/s\n'; <rc> = 0
>Thread-14107::WARNING::2013-03-17 11:14:02,017::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:14:02,027::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:14:04,550::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 11:14:04,552::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 11:14:04,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:14:04,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00133929 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:14:04,600::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00133929 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 11:14:04,601::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 11:14:04,601::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 11:14:04,603::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 11:14:04,604::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:14:04,605::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:14:04,606::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:14:04,606::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:14:04,606::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:14:04,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:14:05,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:14:05,070::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:14:05,070::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 11:14:05,073::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 11:14:05,074::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 11:14:05,075::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 11:14:05,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:14:05,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:14:05,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:14:05,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0448401 s, 91.3 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:14:12,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:14:12,045::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:14:12,076::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:14:12,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00909387 s, 450 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:14:12,523::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:14:12,528::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:14:12,529::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:14:15,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:14:15,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186032 s, 220 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:14:22,156::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:14:22,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000914831 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:14:25,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:14:25,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00747364 s, 548 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:14:32,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:14:32,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771268 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:14:35,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:14:35,627::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911849 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:14:42,286::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 11:14:42,288::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 11:14:42,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:14:42,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0015973 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:14:42,335::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0015973 s, 1.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 11:14:42,336::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 11:14:42,336::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 11:14:42,338::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 11:14:42,339::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:14:42,339::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:14:42,340::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:14:42,340::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:14:42,342::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:14:42,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:14:42,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:14:42,780::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:14:42,781::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 11:14:42,783::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 11:14:42,785::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 11:14:42,785::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 11:14:42,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:14:43,175::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:14:43,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:14:43,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00815123 s, 503 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:14:45,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:14:45,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.046092 s, 88.9 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:14:53,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:14:53,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015009 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:14:55,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:14:55,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737941 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:15:03,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:15:03,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748701 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:15:05,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:15:05,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848364 s, 4.8 MB/s\n'; <rc> = 0
>Thread-14108::WARNING::2013-03-17 11:15:12,552::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:15:12,556::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 11:15:13,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:15:13,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798234 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:15:15,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:15:15,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932625 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:15:22,558::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:15:22,559::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:15:22,978::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:15:22,985::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:15:22,986::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:15:23,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:15:23,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146047 s, 280 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:15:25,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:15:25,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103289 s, 397 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:15:33,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:15:33,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00999994 s, 410 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:15:35,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:15:35,950::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139812 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:15:43,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:15:43,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150259 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:15:45,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:15:46,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000908362 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:15:53,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:15:53,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164833 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:15:56,021::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:15:56,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139736 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:16:03,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:16:03,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169894 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:16:06,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:16:06,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780142 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:16:13,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:16:13,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104963 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:16:16,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:16:16,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175046 s, 234 kB/s\n'; <rc> = 0
>Thread-14109::WARNING::2013-03-17 11:16:23,015::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:16:23,020::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 11:16:23,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:16:24,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772014 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:16:26,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:16:26,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844657 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:16:33,024::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:16:33,025::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:16:33,427::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:16:33,434::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:16:33,435::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:16:34,060::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:16:34,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159883 s, 256 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:16:36,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:16:36,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121929 s, 336 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:16:44,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:16:44,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110012 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:16:46,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:16:46,350::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00244396 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:16:54,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:16:54,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092722 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:16:56,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:16:56,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000702183 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:17:04,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:17:04,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174316 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:17:06,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:17:06,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145803 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:17:14,372::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:17:14,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759751 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:17:16,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:17:16,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778376 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:17:24,439::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:17:24,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152858 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:17:26,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:17:26,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829499 s, 4.9 MB/s\n'; <rc> = 0
>Thread-14110::WARNING::2013-03-17 11:17:33,465::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:17:33,468::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 11:17:34,530::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:17:34,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855909 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:17:36,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:17:36,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785941 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:17:43,470::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:17:43,471::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:17:43,914::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:17:43,921::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:17:43,922::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:17:44,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:17:44,631::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136368 s, 300 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:17:46,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:17:46,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116951 s, 350 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:17:54,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:17:54,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860526 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:17:56,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:17:56,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832773 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:18:04,741::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:18:04,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164422 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:18:06,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:18:06,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000903113 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:18:14,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:18:14,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748253 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:18:16,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:18:16,739::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148121 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:18:24,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:18:24,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135105 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:18:26,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:18:26,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0468389 s, 87.4 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:18:34,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:18:35,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177524 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:18:36,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:18:36,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124914 s, 3.3 MB/s\n'; <rc> = 0
>Thread-14111::WARNING::2013-03-17 11:18:43,958::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:18:43,964::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 11:18:45,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:18:45,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813654 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:18:46,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:18:46,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000934645 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:18:53,966::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 11:18:53,967::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:18:53,968::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:18:54,390::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:18:54,396::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:18:54,398::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:18:55,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:18:55,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123796 s, 331 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:18:56,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:18:57,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163488 s, 251 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:19:05,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:19:05,260::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812861 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:19:07,005::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 11:19:07,013::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 11:19:07,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:19:07,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00135206 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:19:07,058::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00135206 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 11:19:07,058::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 11:19:07,058::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 11:19:07,060::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 11:19:07,061::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:19:07,061::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:19:07,062::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:19:07,062::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:19:07,063::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:19:07,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:19:07,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:19:07,482::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:19:07,483::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 11:19:07,485::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 11:19:07,487::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 11:19:07,488::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 11:19:07,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:19:07,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:19:07,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:19:07,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114611 s, 357 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:19:15,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:19:15,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820352 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:19:17,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:19:18,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152616 s, 268 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:19:25,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:19:25,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899197 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:19:28,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:19:28,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185841 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:19:35,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:19:35,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774841 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:19:38,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:19:38,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140323 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:19:45,501::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 11:19:45,503::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 11:19:45,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:19:45,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000934773 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:19:45,534::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000934773 s, 2.2 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 11:19:45,535::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 11:19:45,535::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 11:19:45,537::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 11:19:45,538::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:19:45,539::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:19:45,539::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:19:45,539::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:19:45,540::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:19:45,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:19:45,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:19:45,992::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:19:45,993::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 11:19:45,995::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 11:19:45,996::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 11:19:45,997::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 11:19:45,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:19:46,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:19:46,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:19:46,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00407152 s, 1.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:19:48,099::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:19:48,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137336 s, 298 kB/s\n'; <rc> = 0
>Thread-14112::WARNING::2013-03-17 11:19:54,424::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:19:54,427::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 11:19:56,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:19:56,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878313 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:19:58,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:19:58,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879318 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:20:04,430::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:20:04,431::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:20:04,857::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:20:04,864::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:20:04,865::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:20:06,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:20:06,561::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131317 s, 312 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:20:08,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:20:08,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184611 s, 222 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:20:16,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:20:16,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157328 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:20:18,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:20:18,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016573 s, 247 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:20:26,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:20:26,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741557 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:20:28,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:20:28,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016317 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:20:36,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:20:36,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150635 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:20:38,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:20:38,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0281477 s, 146 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:20:46,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:20:46,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167843 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:20:48,396::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:20:48,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137014 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:20:56,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:20:56,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143416 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:20:58,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:20:58,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152678 s, 268 kB/s\n'; <rc> = 0
>Thread-14113::WARNING::2013-03-17 11:21:04,892::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:21:04,896::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 11:21:07,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:21:07,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771811 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:21:08,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:21:08,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830814 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:21:14,898::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:21:14,899::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:21:15,421::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:21:15,428::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:21:15,430::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:21:17,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:21:17,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162056 s, 253 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:21:18,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:21:18,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151962 s, 270 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:21:27,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:21:27,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795026 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:21:28,596::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:21:28,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000966688 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:21:37,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:21:37,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150189 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:21:38,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:21:38,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00585461 s, 700 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:21:47,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:21:47,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875708 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:21:48,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:21:48,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143132 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:21:57,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:21:57,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148195 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:21:58,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:21:58,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114306 s, 358 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:22:07,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:22:07,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145739 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:22:08,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:22:08,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00290717 s, 1.4 MB/s\n'; <rc> = 0
>Thread-14114::WARNING::2013-03-17 11:22:15,487::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:22:15,492::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 11:22:17,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:22:17,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00186522 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:22:18,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:22:18,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00492444 s, 832 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:22:25,494::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:22:25,495::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:22:26,140::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:22:26,147::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:22:26,148::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:22:27,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:22:27,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812283 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:22:28,854::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:22:28,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133934 s, 306 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:22:37,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:22:37,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00468752 s, 874 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:22:38,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:22:38,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011401 s, 359 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:22:47,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:22:47,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826771 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:22:48,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:22:48,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143131 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:22:57,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:22:57,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135358 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:22:58,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:22:59,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114242 s, 359 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:23:07,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:23:07,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144326 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:23:09,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:23:09,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748419 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:23:18,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:23:18,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748405 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:23:19,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:23:19,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144529 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14115::WARNING::2013-03-17 11:23:26,175::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:23:26,181::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 11:23:28,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:23:28,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140069 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:23:29,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:23:29,163::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796299 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:23:36,185::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:23:36,186::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:23:36,662::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:23:36,668::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:23:36,670::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:23:38,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:23:38,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137079 s, 299 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:23:39,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:23:39,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164387 s, 249 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:23:48,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:23:48,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802128 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:23:49,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:23:49,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150956 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:23:58,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:23:58,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103578 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:23:59,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:23:59,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880132 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:24:08,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:24:08,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101857 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:24:09,297::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 11:24:09,299::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 11:24:09,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:24:09,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0011776 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:24:09,341::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0011776 s, 1.7 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 11:24:09,342::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 11:24:09,342::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 11:24:09,344::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 11:24:09,345::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:24:09,345::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:24:09,346::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:24:09,346::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:24:09,347::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:24:09,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:24:09,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:24:09,783::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:24:09,783::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 11:24:09,786::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 11:24:09,787::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 11:24:09,788::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 11:24:09,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:24:10,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:24:10,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:24:10,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759538 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:24:18,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:24:18,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923762 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:24:20,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:24:20,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136989 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:24:28,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:24:28,617::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156685 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:24:30,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:24:30,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015565 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14116::WARNING::2013-03-17 11:24:36,705::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:24:36,711::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 11:24:38,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:24:38,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019112 s, 214 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:24:40,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:24:40,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787987 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:24:46,714::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 11:24:46,715::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:24:46,717::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:24:47,187::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:24:47,193::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:24:47,195::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:24:48,742::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 11:24:48,745::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 11:24:48,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:24:48,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00140945 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:24:48,785::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00140945 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 11:24:48,785::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 11:24:48,785::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 11:24:48,787::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 11:24:48,788::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:24:48,789::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:24:48,789::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:24:48,790::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:24:48,790::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:24:48,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:24:49,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:24:49,158::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:24:49,159::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 11:24:49,162::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 11:24:49,163::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 11:24:49,163::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 11:24:49,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:24:49,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:24:49,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:24:49,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109516 s, 374 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:24:50,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:24:50,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00606991 s, 675 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:24:59,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:24:59,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845788 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:25:00,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:25:00,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896748 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:25:09,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:25:09,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115167 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:25:10,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:25:10,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141203 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:25:19,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:25:19,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155782 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:25:20,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:25:20,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126736 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:25:29,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:25:29,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172427 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:25:30,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:25:30,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864563 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:25:39,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:25:39,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146516 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:25:40,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:25:40,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146377 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14117::WARNING::2013-03-17 11:25:47,222::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:25:47,226::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 11:25:50,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:25:50,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811351 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:25:50,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:25:50,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895372 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:25:57,228::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:25:57,230::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:25:57,711::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:25:57,717::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:25:57,718::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:26:00,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:26:00,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000958877 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:26:00,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:26:00,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127594 s, 321 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:26:10,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:26:10,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169789 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:26:10,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:26:10,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151509 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:26:20,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:26:20,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000982891 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:26:20,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:26:20,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107834 s, 380 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:26:30,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:26:30,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792011 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:26:30,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:26:30,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00182977 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:26:40,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:26:40,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111752 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:26:40,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:26:40,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086306 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:26:50,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:26:50,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925991 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:26:50,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:26:50,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193579 s, 212 kB/s\n'; <rc> = 0
>Thread-14118::WARNING::2013-03-17 11:26:57,752::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:26:57,755::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 11:27:00,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:27:00,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000993462 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:27:00,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:27:00,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834619 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:27:07,762::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:27:07,763::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:27:08,296::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:27:08,303::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:27:08,305::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:27:10,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:27:10,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913934 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:27:10,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:27:10,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013527 s, 303 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:27:20,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:27:20,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843384 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:27:20,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:27:20,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166816 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:27:30,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:27:30,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0031533 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:27:30,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:27:31,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887059 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:27:40,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:27:40,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115421 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:27:41,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:27:41,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176465 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:27:50,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:27:50,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801405 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:27:51,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:27:51,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156014 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:28:00,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:28:00,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893817 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:28:01,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:28:01,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768526 s, 5.3 MB/s\n'; <rc> = 0
>Thread-14119::WARNING::2013-03-17 11:28:08,328::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:28:08,332::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 11:28:11,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:28:11,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185639 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:28:11,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:28:11,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808383 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:28:18,334::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:28:18,335::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:28:18,830::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:28:18,838::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:28:18,840::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:28:21,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:28:21,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000962278 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:28:21,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:28:21,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135503 s, 302 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:28:31,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:28:31,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147196 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:28:31,230::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:28:31,297::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000984387 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:28:41,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:28:41,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:28:41,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000960504 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:28:41,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156343 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:28:51,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:28:51,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:28:51,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168608 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:28:51,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135807 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:29:01,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:29:01,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915543 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:29:01,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:29:01,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014926 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:29:11,600::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 11:29:11,602::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 11:29:11,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:29:11,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00137717 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:29:11,643::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00137717 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 11:29:11,643::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 11:29:11,643::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 11:29:11,645::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 11:29:11,649::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:29:11,649::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:29:11,650::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:29:11,650::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:29:11,651::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:29:11,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:29:11,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:29:11,733::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148286 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:29:12,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:29:12,118::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:29:12,119::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 11:29:12,123::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 11:29:12,124::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 11:29:12,125::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 11:29:12,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:29:12,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:29:12,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:29:12,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147638 s, 277 kB/s\n'; <rc> = 0
>Thread-14120::WARNING::2013-03-17 11:29:18,869::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:29:18,874::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 11:29:21,782::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:29:21,820::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114601 s, 357 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:29:22,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:29:22,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000941567 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:29:28,884::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:29:28,886::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:29:29,380::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:29:29,386::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:29:29,388::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:29:31,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:29:31,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148613 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:29:32,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:29:32,611::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116513 s, 352 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:29:41,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:29:41,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153688 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:29:42,617::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:29:42,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091336 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:29:52,026::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 11:29:52,028::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 11:29:52,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:29:52,054::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000761892 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:29:52,056::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000761892 s, 2.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 11:29:52,056::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 11:29:52,057::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 11:29:52,059::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 11:29:52,060::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:29:52,060::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:29:52,061::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:29:52,061::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:29:52,062::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:29:52,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:29:52,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:29:52,476::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:29:52,477::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 11:29:52,479::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 11:29:52,481::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 11:29:52,481::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 11:29:52,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:29:52,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:29:52,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0201059 s, 204 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:29:52,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:29:52,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:29:52,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110695 s, 370 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:30:02,740::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:30:02,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128297 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:30:02,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:30:02,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160877 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:30:12,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:30:12,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132421 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:30:13,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:30:13,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163618 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:30:22,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:30:22,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148127 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:30:23,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:30:23,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166582 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14121::WARNING::2013-03-17 11:30:29,413::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:30:29,416::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:30:32,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:30:32,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809813 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:30:33,178::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:30:33,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818425 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:30:39,418::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 11:30:39,419::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:30:39,420::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:30:39,827::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:30:39,834::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:30:39,835::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:30:42,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:30:42,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0157535 s, 260 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:30:43,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:30:43,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083269 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:30:52,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:30:53,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00968114 s, 423 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:30:53,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:30:53,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014622 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:31:03,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:31:03,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00299402 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:31:03,412::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:31:03,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870035 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:31:13,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:31:13,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113085 s, 362 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:31:13,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:31:13,521::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00221965 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:31:23,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:31:23,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152245 s, 269 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:31:23,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:31:23,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763329 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:31:33,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:31:33,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172081 s, 238 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:31:33,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:31:33,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165727 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14122::WARNING::2013-03-17 11:31:39,863::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:31:39,866::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:31:43,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:31:43,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133343 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:31:43,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:31:43,766::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00885285 s, 463 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:31:49,881::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:31:49,883::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:31:50,297::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:31:50,304::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:31:50,305::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:31:53,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:31:53,372::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00905313 s, 452 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:31:53,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:31:53,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146825 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:32:03,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:32:03,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154196 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:32:03,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:32:03,928::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081081 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:32:13,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:32:13,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167315 s, 245 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:32:13,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:32:14,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170814 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:32:23,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:32:23,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841363 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:32:24,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:32:24,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118095 s, 347 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:32:33,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:32:33,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136358 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:32:34,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:32:34,175::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000698876 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:32:43,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:32:43,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00738901 s, 554 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:32:44,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:32:44,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150695 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14123::WARNING::2013-03-17 11:32:50,332::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:32:50,336::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:32:53,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:32:53,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000929796 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:32:54,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:32:54,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0019854 s, 2.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:33:00,338::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:33:00,339::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:33:00,779::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:33:00,786::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:33:00,787::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:33:03,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:33:03,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143679 s, 285 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:33:04,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:33:04,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085115 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:33:13,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:33:13,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800702 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:33:14,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:33:14,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00302352 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:33:23,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:33:23,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133048 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:33:24,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:33:24,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151989 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:33:33,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:33:33,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128782 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:33:34,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:33:34,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158471 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:33:43,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:33:43,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143386 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:33:44,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:33:44,698::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829116 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:33:53,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:33:53,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769988 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:33:54,738::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:33:54,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137027 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14124::WARNING::2013-03-17 11:34:00,816::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:34:00,820::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:34:03,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:34:03,963::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088959 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:34:04,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:34:04,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909511 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:34:10,822::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:34:10,823::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:34:11,266::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:34:11,273::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:34:11,274::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:34:13,968::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 11:34:13,971::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 11:34:13,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:34:14,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0158079 s, 130 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:34:14,053::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0158079 s, 130 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 11:34:14,053::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 11:34:14,053::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 11:34:14,055::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 11:34:14,056::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:34:14,056::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:34:14,057::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:34:14,057::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:34:14,057::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:34:14,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:34:14,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:34:14,489::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:34:14,490::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 11:34:14,493::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 11:34:14,494::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 11:34:14,495::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 11:34:14,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:34:14,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:34:14,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:34:14,881::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00829914 s, 494 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:34:14,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:34:14,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148815 s, 275 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:34:24,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:34:24,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080001 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:34:24,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:34:24,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813625 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:34:34,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:34:34,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138688 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:34:35,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:34:35,081::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00230397 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:34:44,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:34:45,017::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014269 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:34:45,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:34:45,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145567 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:34:55,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:34:55,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821031 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:34:55,200::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 11:34:55,202::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 11:34:55,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:34:55,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0012458 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:34:55,241::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0012458 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 11:34:55,242::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 11:34:55,242::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 11:34:55,244::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 11:34:55,245::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:34:55,246::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:34:55,246::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:34:55,246::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:34:55,247::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:34:55,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:34:55,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:34:55,695::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:34:55,696::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 11:34:55,699::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 11:34:55,700::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 11:34:55,700::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 11:34:55,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:34:56,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:34:56,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:34:56,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104599 s, 392 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:35:05,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:35:05,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779559 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:35:06,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:35:06,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000954568 s, 4.3 MB/s\n'; <rc> = 0
>Thread-14125::WARNING::2013-03-17 11:35:11,301::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:35:11,305::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:35:15,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:35:15,132::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783458 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:35:16,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:35:16,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846936 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:35:21,307::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:35:21,308::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:35:21,747::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:35:21,756::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:35:21,758::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:35:25,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:35:25,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0420593 s, 97.4 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:35:26,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:35:26,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000931518 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:35:35,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:35:35,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764919 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:35:36,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:35:36,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103026 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:35:45,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:35:45,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137658 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:35:46,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:35:46,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166182 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:35:55,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:35:55,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843507 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:35:56,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:35:56,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837287 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:36:05,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:36:05,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155548 s, 263 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:36:06,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:36:06,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015697 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:36:15,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:36:15,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108109 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:36:16,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:36:16,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00191426 s, 2.1 MB/s\n'; <rc> = 0
>Thread-14126::WARNING::2013-03-17 11:36:21,788::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:36:21,793::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:36:25,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:36:25,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102569 s, 399 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:36:26,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:36:26,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000680448 s, 6.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:36:31,795::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 11:36:31,797::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:36:31,798::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:36:32,193::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:36:32,200::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:36:32,202::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:36:35,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:36:35,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165569 s, 247 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:36:36,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:36:36,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194225 s, 211 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:36:45,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:36:45,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188962 s, 217 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:36:46,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:36:47,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001333 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:36:55,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:36:55,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856362 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:36:57,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:36:57,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861598 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:37:05,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:37:05,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0212914 s, 192 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:37:07,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:37:07,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000948182 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:37:15,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:37:15,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000724319 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:37:17,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:37:17,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838761 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:37:25,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:37:25,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113607 s, 361 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:37:27,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:37:27,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762414 s, 5.4 MB/s\n'; <rc> = 0
>Thread-14127::WARNING::2013-03-17 11:37:32,230::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:37:32,233::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:37:35,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:37:35,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000994805 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:37:37,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:37:37,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779501 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:37:42,235::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:37:42,236::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:37:42,676::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:37:42,682::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:37:42,684::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:37:45,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:37:45,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137793 s, 297 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:37:47,416::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:37:47,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103525 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:37:55,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:37:55,944::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008113 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:37:57,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:37:57,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791568 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:38:05,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:38:05,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00070691 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:38:07,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:38:07,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00183789 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:38:15,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:38:16,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00386606 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:38:17,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:38:17,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810892 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:38:26,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:38:26,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0202516 s, 202 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:38:27,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:38:27,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086945 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:38:36,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:38:36,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000993986 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:38:37,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:38:37,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155048 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14128::WARNING::2013-03-17 11:38:42,710::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:38:42,714::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:38:46,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:38:46,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901787 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:38:47,871::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:38:47,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932875 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:38:52,722::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:38:52,723::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:38:53,138::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:38:53,145::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:38:53,146::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:38:56,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:38:56,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160765 s, 255 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:38:57,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:38:57,963::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758955 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:39:06,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:39:06,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000926245 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:39:08,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:39:08,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00180611 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:39:16,276::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 11:39:16,277::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 11:39:16,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:39:16,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000780198 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:39:16,306::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000780198 s, 2.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 11:39:16,308::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 11:39:16,311::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 11:39:16,313::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 11:39:16,314::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:39:16,314::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:39:16,315::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:39:16,315::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:39:16,316::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:39:16,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:39:16,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:39:16,753::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:39:16,754::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 11:39:16,757::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 11:39:16,758::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 11:39:16,758::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 11:39:16,760::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:39:17,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:39:17,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:39:17,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134665 s, 304 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:39:18,092::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:39:18,130::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0083546 s, 490 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:39:27,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:39:27,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737556 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:39:28,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:39:28,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0019715 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:39:37,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:39:37,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789307 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:39:38,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:39:38,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151153 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:39:47,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:39:47,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012168 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:39:48,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:39:48,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135125 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14129::WARNING::2013-03-17 11:39:53,181::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:39:53,186::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:39:57,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:39:57,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831693 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:39:58,410::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 11:39:58,412::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 11:39:58,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:39:58,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000757147 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:39:58,456::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000757147 s, 2.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 11:39:58,457::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 11:39:58,457::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 11:39:58,458::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 11:39:58,459::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:39:58,460::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:39:58,461::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:39:58,461::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:39:58,461::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:39:58,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:39:58,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:39:58,875::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:39:58,876::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 11:39:58,878::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 11:39:58,879::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 11:39:58,880::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 11:39:58,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:39:59,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:39:59,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:39:59,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00907303 s, 451 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:40:03,189::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:40:03,190::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:40:03,621::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:40:03,627::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:40:03,629::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:40:07,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:40:07,372::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0214625 s, 191 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:40:09,312::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:40:09,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164122 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:40:17,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:40:17,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145749 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:40:19,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:40:19,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867073 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:40:27,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:40:27,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148622 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:40:29,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:40:29,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00201008 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:40:37,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:40:37,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000973445 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:40:39,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:40:39,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134884 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:40:47,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:40:47,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014143 s, 290 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:40:49,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:40:49,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092458 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:40:57,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:40:57,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842375 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:40:59,740::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:40:59,786::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156566 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14130::WARNING::2013-03-17 11:41:03,656::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:41:03,660::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:41:07,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:41:07,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852875 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:41:09,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:41:09,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857335 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:41:13,668::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:41:13,669::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:41:14,102::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:41:14,109::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:41:14,110::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:41:17,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:41:17,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121902 s, 336 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:41:19,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:41:19,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000966991 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:41:27,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:41:27,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0368534 s, 111 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:41:29,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:41:30,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788759 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:41:37,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:41:37,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816841 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:41:40,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:41:40,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849494 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:41:47,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:41:47,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148669 s, 276 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:41:50,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:41:50,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00208617 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:41:57,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:41:57,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842871 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:42:00,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:42:00,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885404 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:42:07,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:42:07,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865137 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:42:10,260::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:42:10,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146415 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14131::WARNING::2013-03-17 11:42:14,137::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:42:14,141::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:42:17,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:42:18,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081246 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:42:20,350::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:42:20,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161132 s, 2.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:42:24,142::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 11:42:24,143::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:42:24,144::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:42:24,602::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:42:24,609::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:42:24,610::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:42:28,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:42:28,040::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0041052 s, 998 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:42:30,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:42:30,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902742 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:42:38,045::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:42:38,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000993669 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:42:40,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:42:40,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158492 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:42:48,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:42:48,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00365181 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:42:50,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:42:50,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152149 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:42:58,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:42:58,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857315 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:43:00,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:43:00,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0226829 s, 181 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:43:08,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:43:08,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808323 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:43:10,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:43:10,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155986 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:43:18,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:43:18,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880828 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:43:20,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:43:20,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165678 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14132::WARNING::2013-03-17 11:43:24,640::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:43:24,642::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:43:28,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:43:28,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178046 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:43:30,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:43:30,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880815 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:43:34,644::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:43:34,645::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:43:35,073::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:43:35,080::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:43:35,081::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:43:38,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:43:38,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144388 s, 284 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:43:40,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:43:40,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833432 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:43:48,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:43:48,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113711 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:43:51,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:43:51,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890592 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:43:58,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:43:58,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087261 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:44:01,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:44:01,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149577 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:44:08,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:44:08,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165446 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:44:11,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:44:11,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152656 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:44:18,479::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 11:44:18,480::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 11:44:18,481::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:44:18,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00118141 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:44:18,521::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00118141 s, 1.7 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 11:44:18,522::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 11:44:18,522::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 11:44:18,524::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 11:44:18,525::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:44:18,525::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:44:18,526::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:44:18,526::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:44:18,527::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:44:18,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:44:18,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:44:18,941::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:44:18,942::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 11:44:18,945::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 11:44:18,946::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 11:44:18,946::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 11:44:18,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:44:19,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:44:19,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:44:19,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154644 s, 265 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:44:21,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:44:21,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853302 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:44:29,345::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:44:29,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00213566 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:44:31,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:44:31,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144984 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14133::WARNING::2013-03-17 11:44:35,114::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:44:35,119::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:44:39,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:44:39,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808677 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:44:41,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:44:41,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909018 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:44:45,122::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:44:45,123::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:44:45,578::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:44:45,585::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:44:45,586::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:44:49,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:44:49,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010943 s, 374 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:44:51,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:44:51,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913181 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:44:59,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:44:59,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820909 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:45:01,765::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 11:45:01,766::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 11:45:01,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:45:01,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00163562 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:45:01,813::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00163562 s, 1.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 11:45:01,814::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 11:45:01,814::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 11:45:01,816::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 11:45:01,817::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:45:01,817::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:45:01,818::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:45:01,818::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:45:01,818::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:45:01,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:45:02,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:45:02,250::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:45:02,251::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 11:45:02,253::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 11:45:02,255::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 11:45:02,255::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 11:45:02,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:45:02,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:45:02,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:45:02,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141669 s, 289 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:45:09,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:45:09,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193831 s, 211 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:45:12,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:45:12,786::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785778 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:45:19,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:45:19,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183116 s, 224 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:45:22,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:45:22,853::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000942238 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:45:29,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:45:29,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159697 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:45:32,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:45:32,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145302 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:45:39,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:45:39,734::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162685 s, 252 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:45:42,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:45:43,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000716306 s, 5.7 MB/s\n'; <rc> = 0
>Thread-14134::WARNING::2013-03-17 11:45:45,609::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:45:45,613::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:45:49,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:45:49,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817744 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:45:53,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:45:53,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835375 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:45:55,669::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:45:55,671::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:45:56,079::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:45:56,086::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:45:56,087::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:45:59,778::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:45:59,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018303 s, 224 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:46:03,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:46:03,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010109 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:46:09,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:46:09,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151708 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:46:13,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:46:13,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915716 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:46:19,875::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:46:19,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000999908 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:46:23,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:46:23,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151916 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:46:29,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:46:29,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766306 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:46:33,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:46:33,382::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753686 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:46:39,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:46:39,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904833 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:46:43,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:46:43,450::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174992 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:46:49,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:46:50,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128484 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:46:53,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:46:53,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125043 s, 3.3 MB/s\n'; <rc> = 0
>Thread-14135::WARNING::2013-03-17 11:46:56,112::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:46:56,116::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:47:00,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:47:00,054::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814761 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:47:03,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:47:03,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185279 s, 2.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:47:06,119::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:47:06,120::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:47:06,547::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:47:06,553::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:47:06,554::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:47:10,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:47:10,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183457 s, 223 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:47:13,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:47:13,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136811 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:47:20,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:47:20,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831438 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:47:23,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:47:23,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146912 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:47:30,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:47:30,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810698 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:47:33,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:47:33,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158015 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:47:40,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:47:40,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776483 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:47:43,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:47:43,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000682939 s, 6.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:47:50,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:47:50,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138911 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:47:53,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:47:54,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139786 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:48:00,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:48:00,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000950778 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:48:04,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:48:04,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141272 s, 290 kB/s\n'; <rc> = 0
>Thread-14136::WARNING::2013-03-17 11:48:06,585::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:48:06,592::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:48:10,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:48:10,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103035 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:48:14,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:48:14,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000733438 s, 5.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:48:16,604::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 11:48:16,605::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:48:16,606::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:48:17,051::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:48:17,057::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:48:17,058::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:48:20,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:48:20,409::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174017 s, 235 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:48:24,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:48:24,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00891909 s, 459 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:48:30,414::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:48:30,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793381 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:48:34,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:48:34,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857345 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:48:40,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:48:40,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896536 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:48:44,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:48:44,398::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764143 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:48:50,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:48:50,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824271 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:48:54,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:48:54,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790897 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:49:00,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:49:00,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868527 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:49:04,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:49:04,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843654 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:49:10,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:49:10,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00236758 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:49:14,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:49:14,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113053 s, 3.6 MB/s\n'; <rc> = 0
>Thread-14137::WARNING::2013-03-17 11:49:17,093::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:49:17,099::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:49:20,595::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 11:49:20,596::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 11:49:20,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:49:20,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000892112 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:49:20,626::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000892112 s, 2.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 11:49:20,626::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 11:49:20,626::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 11:49:20,628::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 11:49:20,629::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:49:20,630::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:49:20,630::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:49:20,631::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:49:20,631::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:49:20,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:49:21,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:49:21,035::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:49:21,036::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 11:49:21,038::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 11:49:21,040::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 11:49:21,040::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 11:49:21,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:49:21,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:49:21,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:49:21,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01601 s, 256 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:49:24,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:49:24,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091521 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:49:27,109::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:49:27,110::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:49:27,559::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:49:27,566::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:49:27,567::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:49:31,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:49:31,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00689419 s, 594 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:49:34,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:49:34,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010886 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:49:41,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:49:41,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805407 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:49:44,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:49:44,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849122 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:49:51,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:49:51,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143101 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:49:54,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:49:54,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867547 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:50:01,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:50:01,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854955 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:50:04,951::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 11:50:04,953::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 11:50:04,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:50:05,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00129811 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:50:05,025::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00129811 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 11:50:05,026::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 11:50:05,026::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 11:50:05,027::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 11:50:05,028::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:50:05,029::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:50:05,030::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:50:05,030::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:50:05,030::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:50:05,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:50:05,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:50:05,460::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:50:05,461::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 11:50:05,465::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 11:50:05,466::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 11:50:05,466::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 11:50:05,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:50:05,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:50:05,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:50:05,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00301715 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:50:11,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:50:11,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140064 s, 292 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:50:15,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:50:15,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145978 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:50:21,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:50:21,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882075 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:50:25,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:50:26,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000983314 s, 4.2 MB/s\n'; <rc> = 0
>Thread-14138::WARNING::2013-03-17 11:50:27,595::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:50:27,599::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:50:31,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:50:31,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863414 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:50:36,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:50:36,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00258077 s, 1.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:50:37,607::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:50:37,608::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:50:38,066::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:50:38,073::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:50:38,074::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:50:41,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:50:41,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00493764 s, 830 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:50:46,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:50:46,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791265 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:50:51,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:50:51,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146613 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:50:56,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:50:56,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000736563 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:51:01,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:51:01,973::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155308 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:51:06,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:51:06,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112842 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:51:11,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:51:12,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800016 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:51:16,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:51:16,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153423 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:51:22,014::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:51:22,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163561 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:51:26,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:51:26,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154947 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:51:32,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:51:32,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144047 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:51:36,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:51:36,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102305 s, 4.0 MB/s\n'; <rc> = 0
>Thread-14139::WARNING::2013-03-17 11:51:38,114::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:51:38,122::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:51:42,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:51:42,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884673 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:51:46,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:51:46,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838545 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:51:48,124::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:51:48,125::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:51:48,534::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:51:48,541::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:51:48,542::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:51:52,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:51:52,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116324 s, 352 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:51:56,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:51:56,742::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0009024 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:52:02,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:52:02,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845942 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:52:06,792::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:52:06,820::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879152 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:52:12,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:52:12,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014362 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:52:16,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:52:16,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000705093 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:52:22,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:52:22,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00671452 s, 610 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:52:26,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:52:26,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909536 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:52:32,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:52:32,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778347 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:52:37,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:52:37,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013924 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:52:42,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:52:42,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853513 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:52:47,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:52:47,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839959 s, 4.9 MB/s\n'; <rc> = 0
>Thread-14140::WARNING::2013-03-17 11:52:48,570::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:52:48,574::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:52:52,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:52:52,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102086 s, 401 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:52:57,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:52:57,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000731664 s, 5.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:52:58,578::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:52:58,580::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:52:59,005::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:52:59,012::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:52:59,013::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:53:02,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:53:02,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129613 s, 316 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:53:07,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:53:07,491::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000750914 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:53:12,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:53:12,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154712 s, 265 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:53:17,530::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:53:17,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737454 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:53:22,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:53:22,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821027 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:53:27,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:53:27,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0419133 s, 97.7 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:53:32,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:53:32,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802909 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:53:37,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:53:37,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012808 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:53:42,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:53:42,631::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828336 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:53:47,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:53:47,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791279 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:53:52,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:53:52,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106312 s, 385 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:53:57,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:53:58,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016152 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14141::WARNING::2013-03-17 11:53:59,038::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:53:59,042::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 11:54:02,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:54:02,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155026 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:54:08,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:54:08,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00188619 s, 2.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:54:09,052::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 11:54:09,053::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:54:09,054::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:54:09,500::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:54:09,507::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:54:09,508::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:54:12,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:54:12,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165977 s, 247 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:54:18,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:54:18,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000934661 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:54:22,786::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 11:54:22,787::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 11:54:22,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:54:22,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000845882 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:54:22,817::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000845882 s, 2.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 11:54:22,817::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 11:54:22,817::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 11:54:22,819::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 11:54:22,820::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:54:22,821::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:54:22,821::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:54:22,821::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:54:22,822::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:54:22,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:54:23,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:54:23,249::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:54:23,250::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 11:54:23,252::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 11:54:23,253::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 11:54:23,254::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 11:54:23,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:54:23,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:54:23,617::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:54:23,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0207075 s, 198 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:54:28,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:54:28,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0027488 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:54:33,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:54:33,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103218 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:54:38,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:54:38,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000730509 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:54:43,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:54:43,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171607 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:54:48,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:54:48,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080839 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:54:53,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:54:53,809::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194402 s, 211 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:54:58,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:54:58,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0265756 s, 154 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:55:03,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:55:03,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789787 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:55:09,009::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 11:55:09,010::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 11:55:09,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:55:09,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.040095 s, 51.1 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:55:09,138::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.040095 s, 51.1 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 11:55:09,140::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 11:55:09,141::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 11:55:09,146::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 11:55:09,152::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:55:09,158::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:55:09,161::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:55:09,164::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:55:09,166::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 11:55:09,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-14142::WARNING::2013-03-17 11:55:09,539::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:55:09,545::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 11:55:09,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:55:09,669::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:55:09,669::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 11:55:09,672::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 11:55:09,673::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 11:55:09,673::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 11:55:09,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:55:10,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:55:10,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:55:10,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00947995 s, 432 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:55:13,854::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:55:13,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151886 s, 270 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:55:19,548::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:55:19,549::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:55:20,000::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:55:20,008::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:55:20,010::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:55:20,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:55:20,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183884 s, 223 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:55:23,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:55:23,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145632 s, 281 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:55:30,221::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:55:30,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000980726 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:55:33,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:55:33,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874539 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:55:40,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:55:40,333::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149522 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:55:43,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:55:44,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013748 s, 298 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:55:50,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:55:50,424::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083233 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:55:54,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:55:54,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139891 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:56:00,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:56:00,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00460674 s, 889 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:56:04,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:56:04,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139956 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:56:10,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:56:10,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842833 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:56:14,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:56:14,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000692625 s, 5.9 MB/s\n'; <rc> = 0
>Thread-14143::WARNING::2013-03-17 11:56:20,043::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:56:20,046::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 11:56:20,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:56:20,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000956444 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:56:24,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:56:24,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825601 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:56:30,049::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:56:30,050::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:56:30,508::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:56:30,517::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:56:30,518::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:56:30,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:56:30,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162568 s, 252 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:56:34,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:56:34,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139911 s, 293 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:56:40,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:56:40,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758719 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:56:44,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:56:44,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0208429 s, 197 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:56:50,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:56:50,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150405 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:56:54,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:56:54,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0230846 s, 177 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:57:00,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:57:00,944::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100119 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:57:04,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:57:04,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076721 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:57:10,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:57:11,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846167 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:57:14,412::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:57:14,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0222978 s, 184 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:57:21,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:57:21,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770011 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:57:24,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:57:24,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0250841 s, 163 kB/s\n'; <rc> = 0
>Thread-14144::WARNING::2013-03-17 11:57:30,544::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:57:30,549::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 11:57:31,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:57:31,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875607 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:57:34,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:57:34,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760868 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:57:40,551::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:57:40,552::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:57:40,992::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:57:40,998::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:57:41,000::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:57:41,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:57:41,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010191 s, 402 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:57:44,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:57:44,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013084 s, 313 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:57:51,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:57:51,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844803 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:57:54,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:57:54,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139784 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:58:01,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:58:01,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148903 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:58:04,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:58:04,709::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00295409 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:58:11,414::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:58:11,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153872 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:58:14,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:58:14,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737904 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:58:21,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:58:21,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151741 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:58:24,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:58:24,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144782 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:58:31,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:58:31,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014409 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:58:34,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:58:34,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780135 s, 5.3 MB/s\n'; <rc> = 0
>Thread-14145::WARNING::2013-03-17 11:58:41,041::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:58:41,044::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 11:58:41,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:58:41,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0270001 s, 152 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:58:44,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:58:44,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149616 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 11:58:51,059::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 11:58:51,060::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 11:58:51,478::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 11:58:51,485::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 11:58:51,486::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 11:58:51,760::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:58:51,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108808 s, 376 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:58:54,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:58:54,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0222955 s, 184 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:59:01,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:59:01,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892553 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:59:04,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:59:05,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0280712 s, 146 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:59:11,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:59:11,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084848 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:59:15,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:59:15,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00303234 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:59:21,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:59:22,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781106 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:59:25,092::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 11:59:25,094::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 11:59:25,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:59:25,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00128893 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:59:25,137::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00128893 s, 1.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 11:59:25,138::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 11:59:25,138::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 11:59:25,140::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 11:59:25,141::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:59:25,143::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:59:25,143::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:59:25,144::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:59:25,144::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 11:59:25,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:59:25,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:59:25,573::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 11:59:25,574::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 11:59:25,576::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 11:59:25,577::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 11:59:25,578::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 11:59:25,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:59:25,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:59:25,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:59:26,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180365 s, 227 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:59:32,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:59:32,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0268425 s, 153 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:59:36,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:59:36,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000722984 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 11:59:42,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:59:42,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000921746 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:59:46,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:59:46,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133256 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14146::WARNING::2013-03-17 11:59:51,513::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 11:59:51,516::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 11:59:52,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 11:59:52,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000938567 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 11:59:56,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 11:59:56,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141095 s, 2.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:00:01,517::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 12:00:01,519::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:00:01,520::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:00:02,183::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:00:02,189::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:00:02,190::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:00:02,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:00:02,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114807 s, 357 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:00:06,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:00:06,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143637 s, 285 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:00:12,405::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 12:00:12,406::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 12:00:12,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:00:12,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00135709 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:00:12,440::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00135709 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 12:00:12,440::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 12:00:12,440::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 12:00:12,442::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 12:00:12,443::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:00:12,444::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:00:12,444::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:00:12,444::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:00:12,445::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:00:12,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:00:12,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:00:12,836::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:00:12,837::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 12:00:12,840::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 12:00:12,841::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 12:00:12,841::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 12:00:12,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:00:13,191::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:00:13,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:00:13,222::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156567 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:00:16,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:00:16,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154006 s, 266 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:00:23,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:00:23,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138791 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:00:26,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:00:26,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785944 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:00:33,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:00:33,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145796 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:00:36,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:00:36,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151298 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:00:43,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:00:43,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156732 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:00:46,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:00:46,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151669 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:00:53,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:00:53,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00183018 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:00:56,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:00:56,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0219242 s, 187 kB/s\n'; <rc> = 0
>Thread-14147::WARNING::2013-03-17 12:01:02,228::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:01:02,241::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:01:03,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:01:03,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752248 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:01:06,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:01:06,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814682 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:01:12,261::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:01:12,262::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:01:12,690::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:01:12,696::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:01:12,698::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:01:13,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:01:13,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0229653 s, 178 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:01:16,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:01:16,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126597 s, 324 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:01:23,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:01:23,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850365 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:01:26,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:01:26,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101682 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:01:33,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:01:33,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137486 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:01:36,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:01:36,700::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110029 s, 372 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:01:43,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:01:44,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157505 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:01:46,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:01:46,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163915 s, 250 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:01:54,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:01:54,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143905 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:01:56,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:01:56,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822317 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:02:04,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:02:04,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136329 s, 300 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:02:06,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:02:06,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791284 s, 5.2 MB/s\n'; <rc> = 0
>Thread-14148::WARNING::2013-03-17 12:02:12,723::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:02:12,727::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:02:14,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:02:14,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817749 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:02:16,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:02:16,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817219 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:02:22,729::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:02:22,730::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:02:23,140::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:02:23,147::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:02:23,148::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:02:24,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:02:24,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018403 s, 223 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:02:26,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:02:26,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159787 s, 256 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:02:34,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:02:34,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841271 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:02:36,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:02:36,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743813 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:02:44,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:02:44,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777727 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:02:46,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:02:46,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000947384 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:02:54,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:02:54,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076452 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:02:56,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:02:57,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152441 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:03:04,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:03:04,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760115 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:03:07,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:03:07,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163761 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:03:14,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:03:14,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147983 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:03:17,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:03:17,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167423 s, 2.4 MB/s\n'; <rc> = 0
>Thread-14149::WARNING::2013-03-17 12:03:23,177::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:03:23,183::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:03:24,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:03:24,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000707417 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:03:27,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:03:27,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000922369 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:03:33,186::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:03:33,188::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:03:33,633::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:03:33,640::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:03:33,642::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:03:34,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:03:34,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131023 s, 313 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:03:37,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:03:37,222::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188673 s, 217 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:03:44,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:03:44,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134079 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:03:47,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:03:47,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780469 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:03:54,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:03:55,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781254 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:03:57,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:03:57,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135491 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:04:05,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:04:05,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135061 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:04:07,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:04:07,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0209232 s, 196 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:04:15,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:04:15,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147682 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:04:17,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:04:17,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.02481 s, 165 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:04:25,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:04:25,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017781 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:04:27,425::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 12:04:27,430::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 12:04:27,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:04:27,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000762121 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:04:27,465::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000762121 s, 2.7 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 12:04:27,466::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 12:04:27,466::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 12:04:27,468::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 12:04:27,470::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:04:27,470::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:04:27,471::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:04:27,471::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:04:27,472::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:04:27,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:04:27,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:04:27,909::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:04:27,910::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 12:04:27,912::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 12:04:27,914::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 12:04:27,914::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 12:04:27,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:04:28,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:04:28,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:04:28,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156612 s, 262 kB/s\n'; <rc> = 0
>Thread-14150::WARNING::2013-03-17 12:04:33,671::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:04:33,677::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:04:35,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:04:35,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848522 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:04:38,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:04:38,366::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085361 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:04:43,681::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:04:43,682::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:04:44,143::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:04:44,149::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:04:44,151::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:04:45,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:04:45,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0282723 s, 145 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:04:48,372::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:04:48,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171496 s, 239 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:04:55,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:04:55,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0278022 s, 147 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:04:58,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:04:58,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101264 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:05:05,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:05:05,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0387659 s, 106 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:05:08,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:05:08,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001861 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:05:15,705::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 12:05:15,706::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 12:05:15,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:05:15,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00118168 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:05:15,747::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00118168 s, 1.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 12:05:15,748::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 12:05:15,748::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 12:05:15,750::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 12:05:15,751::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:05:15,751::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:05:15,752::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:05:15,752::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:05:15,752::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:05:15,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:05:16,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:05:16,159::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:05:16,160::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 12:05:16,162::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 12:05:16,164::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 12:05:16,164::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 12:05:16,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:05:16,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:05:16,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:05:16,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105532 s, 388 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:05:18,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:05:18,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178481 s, 229 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:05:26,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:05:26,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152235 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:05:28,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:05:28,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776922 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:05:36,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:05:36,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154401 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:05:38,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:05:38,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00205488 s, 2.0 MB/s\n'; <rc> = 0
>Thread-14151::WARNING::2013-03-17 12:05:44,183::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:05:44,186::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:05:46,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:05:46,805::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083278 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:05:48,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:05:48,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846151 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:05:54,192::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 12:05:54,193::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:05:54,194::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:05:54,636::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:05:54,643::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:05:54,644::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:05:56,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:05:56,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0231107 s, 177 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:05:58,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:05:58,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0223161 s, 184 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:06:06,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:06:06,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861324 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:06:08,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:06:08,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015477 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:06:17,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:06:17,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119546 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:06:18,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:06:18,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890077 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:06:27,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:06:27,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000722921 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:06:28,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:06:28,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860023 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:06:37,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:06:37,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146297 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:06:38,894::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:06:38,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157189 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:06:47,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:06:47,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166387 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:06:48,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:06:48,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0202265 s, 203 kB/s\n'; <rc> = 0
>Thread-14152::WARNING::2013-03-17 12:06:54,669::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:06:54,672::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:06:57,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:06:57,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831179 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:06:59,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:06:59,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0219729 s, 186 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:07:04,676::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:07:04,678::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:07:05,171::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:07:05,178::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:07:05,179::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:07:07,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:07:07,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.025265 s, 162 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:07:09,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:07:09,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140168 s, 292 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:07:17,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:07:17,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807711 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:07:19,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:07:19,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0232329 s, 176 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:07:27,537::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:07:27,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155508 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:07:29,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:07:29,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835476 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:07:37,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:07:37,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154687 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:07:39,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:07:39,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000745272 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:07:47,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:07:47,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917832 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:07:49,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:07:49,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147907 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:07:57,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:07:57,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146862 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:07:59,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:07:59,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0215235 s, 190 kB/s\n'; <rc> = 0
>Thread-14153::WARNING::2013-03-17 12:08:05,206::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:08:05,211::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:08:07,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:08:07,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148234 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:08:09,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:08:09,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779237 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:08:15,216::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:08:15,217::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:08:15,643::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:08:15,649::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:08:15,651::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:08:17,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:08:17,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190304 s, 215 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:08:19,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:08:19,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.005575 s, 735 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:08:28,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:08:28,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796827 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:08:29,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:08:29,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140635 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:08:38,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:08:38,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764704 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:08:39,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:08:39,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738912 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:08:48,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:08:48,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894329 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:08:49,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:08:49,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819401 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:08:58,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:08:58,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156932 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:08:59,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:08:59,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814421 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:09:08,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:09:08,347::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0268825 s, 152 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:09:09,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:09:09,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158152 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14154::WARNING::2013-03-17 12:09:15,677::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:09:15,681::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:09:18,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:09:18,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800201 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:09:19,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:09:19,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000980711 s, 4.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:09:25,683::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:09:25,684::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:09:26,175::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:09:26,181::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:09:26,183::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:09:28,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:09:28,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0230487 s, 178 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:09:29,668::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 12:09:29,670::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 12:09:29,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:09:29,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00801803 s, 255 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:09:29,707::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00801803 s, 255 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 12:09:29,707::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 12:09:29,708::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 12:09:29,710::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 12:09:29,711::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:09:29,712::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:09:29,712::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:09:29,713::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:09:29,713::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:09:29,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:09:30,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:09:30,172::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:09:30,173::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 12:09:30,182::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 12:09:30,185::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 12:09:30,186::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 12:09:30,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:09:30,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:09:30,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:09:30,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00441877 s, 927 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:09:38,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:09:38,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000961189 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:09:40,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:09:40,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0229461 s, 179 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:09:48,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:09:48,657::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149758 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:09:50,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:09:50,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000995468 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:09:58,697::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:09:58,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152091 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:10:00,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:10:00,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00098427 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:10:08,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:10:08,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767306 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:10:10,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:10:10,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.023654 s, 173 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:10:18,848::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 12:10:18,850::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 12:10:18,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:10:18,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000826094 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:10:18,888::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000826094 s, 2.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 12:10:18,888::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 12:10:18,888::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 12:10:18,890::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 12:10:18,891::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:10:18,892::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:10:18,892::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:10:18,893::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:10:18,893::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:10:18,894::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:10:19,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:10:19,346::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:10:19,346::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 12:10:19,349::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 12:10:19,350::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 12:10:19,350::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 12:10:19,352::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:10:19,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:10:19,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:10:19,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00940939 s, 435 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:10:20,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:10:20,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00900752 s, 455 kB/s\n'; <rc> = 0
>Thread-14155::WARNING::2013-03-17 12:10:26,209::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:10:26,212::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:10:29,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:10:29,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.025432 s, 161 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:10:30,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:10:30,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769283 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:10:36,215::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:10:36,216::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:10:36,695::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:10:36,702::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:10:36,703::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:10:40,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:10:40,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206493 s, 198 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:10:40,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:10:40,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142912 s, 287 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:10:50,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:10:50,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146449 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:10:50,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:10:51,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892608 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:11:00,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:11:00,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0264828 s, 155 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:11:01,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:11:01,054::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142932 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:11:10,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:11:10,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826657 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:11:11,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:11:11,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170169 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:11:20,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:11:20,395::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812294 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:11:21,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:11:21,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148342 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:11:30,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:11:30,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162388 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:11:31,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:11:31,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000929563 s, 4.4 MB/s\n'; <rc> = 0
>Thread-14156::WARNING::2013-03-17 12:11:36,729::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:11:36,733::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:11:40,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:11:40,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000926796 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:11:41,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:11:41,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146771 s, 2.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:11:46,734::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 12:11:46,736::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:11:46,737::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:11:47,361::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:11:47,369::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:11:47,370::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:11:50,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:11:50,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135529 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:11:51,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:11:51,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185818 s, 220 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:12:00,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:12:00,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173722 s, 236 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:12:01,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:12:01,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0215286 s, 190 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:12:10,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:12:10,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898416 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:12:11,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:12:11,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788853 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:12:20,837::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:12:20,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00430486 s, 951 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:12:21,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:12:21,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800292 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:12:30,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:12:30,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152165 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:12:31,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:12:31,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.025278 s, 162 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:12:40,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:12:41,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0269878 s, 152 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:12:41,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:12:41,569::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00202502 s, 2.0 MB/s\n'; <rc> = 0
>Thread-14157::WARNING::2013-03-17 12:12:47,398::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:12:47,402::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:12:51,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:12:51,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847098 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:12:51,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:12:51,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000974116 s, 4.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:12:57,404::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:12:57,405::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:12:57,876::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:12:57,882::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:12:57,884::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:13:01,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:13:01,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0263931 s, 155 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:13:01,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:13:01,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110236 s, 372 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:13:11,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:13:11,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147491 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:13:11,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:13:11,744::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110143 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:13:21,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:13:21,413::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147787 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:13:21,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:13:21,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106474 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:13:31,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:13:31,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083143 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:13:31,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:13:31,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142359 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:13:41,524::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:13:41,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0277394 s, 148 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:13:41,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:13:41,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016233 s, 252 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:13:51,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:13:51,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791407 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:13:51,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:13:51,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853277 s, 4.8 MB/s\n'; <rc> = 0
>Thread-14158::WARNING::2013-03-17 12:13:57,914::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:13:57,918::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:14:01,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:14:01,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164376 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:14:01,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:14:02,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786833 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:14:07,920::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:14:07,923::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:14:08,335::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:14:08,341::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:14:08,343::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:14:11,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:14:11,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907349 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:14:12,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:14:12,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162823 s, 252 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:14:21,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:14:21,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849688 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:14:22,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:14:22,132::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105016 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:14:31,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:14:31,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0321872 s, 127 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:14:32,137::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 12:14:32,141::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 12:14:32,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:14:32,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000856464 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:14:32,172::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000856464 s, 2.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 12:14:32,172::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 12:14:32,173::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 12:14:32,177::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 12:14:32,182::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:14:32,182::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:14:32,183::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:14:32,183::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:14:32,184::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:14:32,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:14:32,611::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:14:32,618::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:14:32,619::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 12:14:32,623::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 12:14:32,624::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 12:14:32,624::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 12:14:32,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:14:32,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:14:32,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:14:33,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138977 s, 295 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:14:42,060::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:14:42,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0256794 s, 160 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:14:43,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:14:43,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015039 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:14:52,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:14:52,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001738 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:14:53,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:14:53,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837936 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:15:02,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:15:02,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740831 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:15:03,129::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:15:03,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133646 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14159::WARNING::2013-03-17 12:15:08,370::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:15:08,373::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:15:12,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:15:12,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000938698 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:15:13,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:15:13,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818725 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:15:18,375::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:15:18,376::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:15:18,815::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:15:18,821::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:15:18,823::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:15:22,393::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 12:15:22,399::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 12:15:22,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:15:22,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000779724 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:15:22,428::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000779724 s, 2.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 12:15:22,428::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 12:15:22,428::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 12:15:22,430::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 12:15:22,431::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:15:22,432::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:15:22,432::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:15:22,433::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:15:22,433::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:15:22,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:15:22,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:15:22,882::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:15:22,883::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 12:15:22,886::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 12:15:22,887::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 12:15:22,887::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 12:15:22,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:15:23,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:15:23,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:15:23,271::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:15:23,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0202254 s, 203 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:15:23,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144563 s, 283 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:15:33,324::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:15:33,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016659 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:15:33,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:15:33,408::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883801 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:15:43,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:15:43,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141806 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:15:43,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:15:43,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137487 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:15:53,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:15:53,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899355 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:15:53,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:15:53,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770169 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:16:03,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:16:03,494::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0047356 s, 865 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:16:03,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:16:03,649::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141911 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:16:13,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:16:13,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014555 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:16:13,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:16:13,733::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178904 s, 2.3 MB/s\n'; <rc> = 0
>Thread-14160::WARNING::2013-03-17 12:16:18,856::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:16:18,863::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 12:16:23,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:16:23,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145563 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:16:23,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:16:23,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000701239 s, 5.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:16:28,866::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:16:28,867::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:16:29,293::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:16:29,299::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:16:29,301::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:16:33,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:16:33,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161051 s, 254 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:16:33,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:16:33,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00187275 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:16:43,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:16:43,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911637 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:16:43,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:16:43,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887544 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:16:53,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:16:53,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158574 s, 258 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:16:53,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:16:54,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856603 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:17:03,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:17:03,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142142 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:17:04,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:17:04,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140687 s, 291 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:17:13,797::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:17:13,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0224874 s, 182 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:17:14,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:17:14,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0231734 s, 177 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:17:23,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:17:23,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.002142 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:17:24,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:17:24,299::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815527 s, 5.0 MB/s\n'; <rc> = 0
>Thread-14161::WARNING::2013-03-17 12:17:29,333::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:17:29,339::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 12:17:33,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:17:33,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0238179 s, 172 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:17:34,350::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:17:34,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000700832 s, 5.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:17:39,342::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 12:17:39,343::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:17:39,344::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:17:39,736::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:17:39,743::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:17:39,744::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:17:43,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:17:44,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0276044 s, 148 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:17:44,416::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:17:44,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00184253 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:17:54,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:17:54,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876091 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:17:54,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:17:54,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015017 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:18:04,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:18:04,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134236 s, 305 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:18:04,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:18:04,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0342909 s, 119 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:18:14,118::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:18:14,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0277072 s, 148 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:18:14,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:18:14,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0279268 s, 147 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:18:24,194::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:18:24,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895418 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:18:24,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:18:24,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132282 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:18:34,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:18:34,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0219859 s, 186 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:18:34,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:18:34,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145046 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14162::WARNING::2013-03-17 12:18:39,770::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:18:39,774::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 12:18:44,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:18:44,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895924 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:18:44,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:18:44,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00207115 s, 2.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:18:49,776::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:18:49,778::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:18:50,194::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:18:50,200::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:18:50,202::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:18:54,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:18:54,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165991 s, 247 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:18:55,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:18:55,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000994443 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:19:04,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:19:04,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777971 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:19:05,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:19:05,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014924 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:19:14,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:19:14,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0253811 s, 161 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:19:15,178::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:19:15,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015482 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:19:24,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:19:24,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139982 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:19:25,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:19:25,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149986 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:19:34,534::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 12:19:34,536::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 12:19:34,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:19:34,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00136467 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:19:34,578::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00136467 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 12:19:34,579::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 12:19:34,579::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 12:19:34,581::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 12:19:34,582::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:19:34,582::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:19:34,583::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:19:34,583::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:19:34,583::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:19:34,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:19:35,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:19:35,006::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:19:35,007::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 12:19:35,010::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 12:19:35,014::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 12:19:35,014::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 12:19:35,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:19:35,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:19:35,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:19:35,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:19:35,439::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147106 s, 278 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:19:35,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123084 s, 333 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:19:45,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:19:45,524::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:19:45,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821958 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:19:45,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000979328 s, 4.2 MB/s\n'; <rc> = 0
>Thread-14163::WARNING::2013-03-17 12:19:50,232::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:19:50,236::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 12:19:55,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:19:55,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817717 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:19:55,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:19:55,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0018877 s, 2.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:20:00,238::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:20:00,239::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:20:00,685::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:20:00,692::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:20:00,693::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:20:05,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:20:05,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:20:05,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00743211 s, 551 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:20:05,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012068 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:20:15,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:20:15,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:20:15,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143768 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:20:15,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156107 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:20:25,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:20:25,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784267 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:20:25,919::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 12:20:25,920::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 12:20:25,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:20:25,958::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00130663 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:20:25,959::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00130663 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 12:20:25,960::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 12:20:25,960::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 12:20:25,962::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 12:20:25,963::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:20:25,964::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:20:25,964::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:20:25,964::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:20:25,965::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:20:25,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:20:26,402::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:20:26,414::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:20:26,415::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 12:20:26,417::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 12:20:26,420::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 12:20:26,421::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 12:20:26,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:20:26,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:20:26,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:20:26,818::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100046 s, 409 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:20:35,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:20:35,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000681059 s, 6.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:20:36,859::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:20:36,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924551 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:20:45,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:20:45,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133317 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:20:46,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:20:46,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757384 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:20:55,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:20:56,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016679 s, 246 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:20:57,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:20:57,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146982 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14164::WARNING::2013-03-17 12:21:00,719::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:21:00,722::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 12:21:06,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:21:06,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0567796 s, 72.1 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:21:07,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:21:07,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869603 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:21:10,725::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:21:10,726::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:21:11,130::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:21:11,137::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:21:11,140::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:21:16,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:21:16,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170374 s, 240 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:21:17,156::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:21:17,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776074 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:21:26,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:21:26,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163142 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:21:27,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:21:27,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101648 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:21:36,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:21:36,350::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123436 s, 332 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:21:37,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:21:37,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084049 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:21:46,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:21:46,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864436 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:21:47,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:21:47,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154915 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:21:56,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:21:56,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0378082 s, 108 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:21:57,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:21:57,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146994 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:22:06,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:22:06,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100845 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:22:07,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:22:07,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.020478 s, 200 kB/s\n'; <rc> = 0
>Thread-14165::WARNING::2013-03-17 12:22:11,170::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:22:11,174::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 12:22:16,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:22:16,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877518 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:22:17,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:22:17,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083963 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:22:21,176::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:22:21,177::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:22:21,597::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:22:21,603::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:22:21,605::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:22:26,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:22:26,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00947309 s, 432 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:22:27,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:22:27,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140181 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:22:36,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:22:36,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838363 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:22:37,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:22:37,866::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859582 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:22:46,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:22:46,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845646 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:22:47,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:22:47,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899504 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:22:56,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:22:56,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146179 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:22:57,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:22:58,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159196 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:23:06,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:23:06,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014143 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:23:08,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:23:08,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010004 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:23:16,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:23:16,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135082 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:23:18,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:23:18,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015321 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14166::WARNING::2013-03-17 12:23:21,640::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:23:21,643::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 12:23:26,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:23:26,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806042 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:23:28,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:23:28,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798666 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:23:31,645::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 12:23:31,646::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:23:31,662::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:23:32,133::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:23:32,139::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:23:32,141::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:23:36,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:23:36,890::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160502 s, 255 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:23:38,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:23:38,409::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859942 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:23:46,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:23:46,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010033 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:23:48,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:23:48,698::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143703 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:23:56,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:23:56,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138533 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:23:58,738::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:23:58,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830531 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:24:06,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:24:07,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779773 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:24:08,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:24:08,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897023 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:24:17,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:24:17,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161301 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:24:18,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:24:18,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000714437 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:24:27,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:24:27,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853351 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:24:28,959::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:24:28,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000943948 s, 4.3 MB/s\n'; <rc> = 0
>Thread-14167::WARNING::2013-03-17 12:24:32,176::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:24:32,180::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 12:24:37,129::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 12:24:37,130::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 12:24:37,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:24:37,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0221497 s, 92.5 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:24:37,181::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0221497 s, 92.5 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 12:24:37,182::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 12:24:37,182::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 12:24:37,184::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 12:24:37,186::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:24:37,187::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:24:37,187::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:24:37,188::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:24:37,188::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:24:37,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:24:37,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:24:37,620::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:24:37,621::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 12:24:37,624::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 12:24:37,625::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 12:24:37,626::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 12:24:37,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:24:37,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:24:37,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:24:38,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00856541 s, 478 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:24:39,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:24:39,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206196 s, 199 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:24:42,183::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:24:42,184::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:24:42,581::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:24:42,588::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:24:42,589::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:24:48,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:24:48,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173928 s, 235 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:24:49,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:24:49,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0305846 s, 134 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:24:58,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:24:58,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855244 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:24:59,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:24:59,287::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.02634 s, 156 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:25:08,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:25:08,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102576 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:25:09,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:25:09,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831495 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:25:18,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:25:18,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138575 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:25:19,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:25:19,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000721371 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:25:28,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:25:28,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767775 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:25:29,532::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 12:25:29,534::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 12:25:29,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:25:29,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000751902 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:25:29,573::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000751902 s, 2.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 12:25:29,574::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 12:25:29,574::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 12:25:29,576::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 12:25:29,577::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:25:29,578::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:25:29,578::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:25:29,578::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:25:29,579::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:25:29,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:25:30,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:25:30,006::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:25:30,007::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 12:25:30,009::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 12:25:30,011::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 12:25:30,012::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 12:25:30,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:25:30,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:25:30,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:25:30,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.008202 s, 499 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:25:38,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:25:38,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181684 s, 225 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:25:40,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:25:40,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011471 s, 3.6 MB/s\n'; <rc> = 0
>Thread-14168::WARNING::2013-03-17 12:25:42,617::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:25:42,619::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 12:25:48,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:25:48,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111749 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:25:50,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:25:50,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772951 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:25:52,622::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:25:52,623::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:25:53,088::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:25:53,095::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:25:53,096::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:25:58,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:25:58,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119038 s, 344 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:26:00,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:26:00,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000903754 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:26:08,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:26:08,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0218063 s, 188 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:26:10,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:26:10,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873818 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:26:18,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:26:18,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738602 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:26:20,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:26:20,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152738 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:26:28,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:26:28,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738108 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:26:30,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:26:30,880::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783773 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:26:38,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:26:38,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145896 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:26:40,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:26:40,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873916 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:26:48,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:26:48,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130631 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:26:50,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:26:51,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158945 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14169::WARNING::2013-03-17 12:26:53,121::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:26:53,125::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 12:26:58,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:26:58,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014565 s, 281 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:27:01,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:27:01,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0338109 s, 121 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:27:03,127::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:27:03,128::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:27:03,588::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:27:03,595::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:27:03,596::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:27:08,730::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:27:08,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113021 s, 362 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:27:11,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:27:11,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173761 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:27:18,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:27:18,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920099 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:27:21,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:27:21,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016073 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:27:28,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:27:28,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149746 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:27:31,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:27:31,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00288226 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:27:38,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:27:38,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010975 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:27:41,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:27:41,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139931 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:27:48,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:27:48,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829521 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:27:51,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:27:51,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014374 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:27:58,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:27:58,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141782 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:28:01,592::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:28:01,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00298122 s, 1.4 MB/s\n'; <rc> = 0
>Thread-14170::WARNING::2013-03-17 12:28:03,625::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:28:03,628::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 12:28:08,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:28:09,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144265 s, 284 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:28:11,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:28:11,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150775 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:28:13,631::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:28:13,632::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:28:14,062::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:28:14,069::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:28:14,070::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:28:19,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:28:19,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119396 s, 343 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:28:21,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:28:21,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865227 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:28:29,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:28:29,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206427 s, 198 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:28:31,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:28:31,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00191789 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:28:39,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:28:39,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112348 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:28:41,993::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:28:42,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167883 s, 244 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:28:49,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:28:49,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180392 s, 227 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:28:52,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:28:52,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167676 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:28:59,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:28:59,309::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132834 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:29:02,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:29:02,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146752 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:29:09,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:29:09,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203306 s, 201 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:29:12,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:29:12,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166887 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14171::WARNING::2013-03-17 12:29:14,105::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:29:14,109::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 12:29:19,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:29:19,409::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913855 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:29:22,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:29:22,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160197 s, 2.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:29:24,112::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 12:29:24,113::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:29:24,114::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:29:24,627::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:29:24,634::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:29:24,635::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:29:29,416::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:29:29,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190014 s, 216 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:29:32,415::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:29:32,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000937165 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:29:39,481::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 12:29:39,482::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 12:29:39,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:29:39,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00129649 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:29:39,529::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00129649 s, 1.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 12:29:39,530::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 12:29:39,530::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 12:29:39,531::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 12:29:39,532::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:29:39,533::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:29:39,534::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:29:39,534::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:29:39,534::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:29:39,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:29:39,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:29:39,982::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:29:39,983::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 12:29:39,986::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 12:29:39,987::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 12:29:39,988::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 12:29:39,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:29:40,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:29:40,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:29:40,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150881 s, 271 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:29:42,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:29:42,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0233506 s, 175 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:29:50,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:29:50,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189875 s, 216 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:29:52,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:29:52,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000696914 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:30:00,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:30:00,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856197 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:30:02,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:30:02,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135975 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:30:10,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:30:10,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180993 s, 226 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:30:12,740::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:30:12,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145078 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:30:20,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:30:20,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014773 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:30:22,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:30:22,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121818 s, 3.4 MB/s\n'; <rc> = 0
>Thread-14172::WARNING::2013-03-17 12:30:24,660::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:30:24,664::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 12:30:30,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:30:30,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131896 s, 311 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:30:32,927::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 12:30:32,928::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 12:30:32,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:30:32,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000989682 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:30:32,956::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000989682 s, 2.1 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 12:30:32,956::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 12:30:32,957::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 12:30:32,958::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 12:30:32,959::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:30:32,960::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:30:32,961::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:30:32,961::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:30:32,961::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:30:32,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:30:33,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:30:33,395::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:30:33,396::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 12:30:33,398::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 12:30:33,400::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 12:30:33,400::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 12:30:33,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:30:33,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:30:33,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:30:33,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00904118 s, 453 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:30:34,666::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:30:34,667::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:30:35,087::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:30:35,093::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:30:35,094::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:30:40,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:30:40,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013381 s, 306 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:30:43,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:30:43,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884727 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:30:50,717::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:30:50,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140375 s, 292 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:30:53,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:30:53,936::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811683 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:31:00,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:31:00,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768767 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:31:03,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:31:04,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138807 s, 295 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:31:10,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:31:10,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165045 s, 248 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:31:14,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:31:14,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846976 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:31:20,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:31:20,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139621 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:31:24,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:31:24,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118941 s, 344 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:31:30,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:31:30,963::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0207593 s, 197 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:31:34,352::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:31:34,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000739213 s, 5.5 MB/s\n'; <rc> = 0
>Thread-14173::WARNING::2013-03-17 12:31:35,125::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:31:35,132::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 12:31:40,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:31:40,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902941 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:31:44,592::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:31:44,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082279 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:31:45,144::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:31:45,145::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:31:45,612::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:31:45,619::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:31:45,620::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:31:51,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:31:51,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0240432 s, 170 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:31:54,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:31:54,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783897 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:32:01,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:32:01,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00653115 s, 627 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:32:04,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:32:04,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163537 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:32:11,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:32:11,132::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081012 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:32:14,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:32:14,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000952847 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:32:21,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:32:21,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170043 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:32:24,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:32:24,992::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790023 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:32:31,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:32:31,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901031 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:32:35,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:32:35,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167042 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:32:41,221::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:32:41,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014415 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:32:45,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:32:45,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153189 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14174::WARNING::2013-03-17 12:32:45,646::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:32:45,650::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 12:32:51,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:32:51,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000918762 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:32:55,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:32:55,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0265288 s, 154 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:32:55,658::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:32:55,660::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:32:56,126::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:32:56,132::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:32:56,133::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:33:01,312::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:33:01,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136893 s, 299 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:33:05,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:33:05,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818609 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:33:11,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:33:11,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738909 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:33:15,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:33:15,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824429 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:33:21,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:33:21,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832755 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:33:25,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:33:25,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830326 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:33:31,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:33:31,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084669 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:33:35,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:33:35,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830324 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:33:41,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:33:41,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00250024 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:33:45,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:33:45,944::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184691 s, 222 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:33:51,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:33:51,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000739593 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:33:55,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:33:56,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0217571 s, 188 kB/s\n'; <rc> = 0
>Thread-14175::WARNING::2013-03-17 12:33:56,164::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:33:56,178::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 12:34:01,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:34:01,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117334 s, 3.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:34:06,182::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:34:06,183::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:34:06,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:34:06,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126876 s, 323 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:34:06,657::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:34:06,663::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:34:06,664::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:34:11,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:34:11,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184907 s, 222 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:34:16,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:34:16,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00072576 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:34:21,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:34:21,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00226622 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:34:26,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:34:26,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873871 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:34:32,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:34:32,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0231902 s, 177 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:34:36,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:34:36,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000942157 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:34:42,112::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 12:34:42,114::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 12:34:42,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:34:42,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00933488 s, 219 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:34:42,194::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00933488 s, 219 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 12:34:42,194::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 12:34:42,194::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 12:34:42,196::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 12:34:42,197::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:34:42,197::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:34:42,198::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:34:42,198::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:34:42,198::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:34:42,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:34:42,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:34:42,634::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:34:42,635::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 12:34:42,640::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 12:34:42,641::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 12:34:42,641::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 12:34:42,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:34:42,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:34:42,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:34:43,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102213 s, 401 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:34:46,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:34:46,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019602 s, 209 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:34:53,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:34:53,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138318 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:34:56,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:34:56,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149305 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:35:03,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:35:03,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00182492 s, 2.2 MB/s\n'; <rc> = 0
>Thread-14176::WARNING::2013-03-17 12:35:06,694::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:35:06,699::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:35:06,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:35:06,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162316 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:35:13,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:35:13,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746724 s, 5.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:35:16,703::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 12:35:16,704::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:35:16,705::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:35:16,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:35:16,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01257 s, 326 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:35:17,159::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:35:17,170::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:35:17,172::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:35:23,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:35:23,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131009 s, 313 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:35:26,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:35:26,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0255872 s, 160 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:35:33,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:35:33,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000716575 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:35:37,005::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 12:35:37,007::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 12:35:37,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:35:37,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00149043 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:35:37,049::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00149043 s, 1.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 12:35:37,049::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 12:35:37,049::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 12:35:37,052::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 12:35:37,053::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:35:37,053::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:35:37,054::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:35:37,054::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:35:37,055::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:35:37,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:35:37,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:35:37,449::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:35:37,450::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 12:35:37,452::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 12:35:37,454::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 12:35:37,454::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 12:35:37,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:35:37,809::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:35:37,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:35:37,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0195448 s, 210 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:35:43,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:35:43,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0360351 s, 114 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:35:47,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:35:47,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877385 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:35:53,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:35:53,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781377 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:35:57,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:35:58,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0231903 s, 177 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:36:03,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:36:03,424::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133617 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:36:08,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:36:08,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118711 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:36:13,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:36:13,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000705709 s, 5.8 MB/s\n'; <rc> = 0
>Thread-14177::WARNING::2013-03-17 12:36:17,216::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:36:17,221::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:36:18,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:36:18,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0211674 s, 194 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:36:23,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:36:23,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780608 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:36:27,225::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:36:27,227::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:36:27,667::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:36:27,674::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:36:27,675::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:36:28,271::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:36:28,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125688 s, 326 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:36:33,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:36:33,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162839 s, 252 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:36:38,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:36:38,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199321 s, 205 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:36:43,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:36:43,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162628 s, 252 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:36:48,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:36:48,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107936 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:36:53,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:36:53,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741419 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:36:58,524::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:36:58,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018025 s, 227 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:37:03,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:37:03,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0239789 s, 171 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:37:08,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:37:08,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813738 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:37:13,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:37:13,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140689 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:37:18,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:37:18,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769392 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:37:23,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:37:23,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0196735 s, 208 kB/s\n'; <rc> = 0
>Thread-14178::WARNING::2013-03-17 12:37:27,704::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:37:27,707::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:37:28,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:37:28,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079351 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:37:33,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:37:33,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831414 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:37:37,710::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:37:37,711::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:37:38,214::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:37:38,220::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:37:38,222::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:37:38,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:37:38,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146732 s, 279 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:37:43,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:37:43,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139167 s, 294 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:37:48,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:37:48,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149638 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:37:53,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:37:54,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181536 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:37:59,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:37:59,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.024993 s, 164 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:38:04,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:38:04,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0257627 s, 159 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:38:09,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:38:09,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.027417 s, 149 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:38:14,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:38:14,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819043 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:38:19,221::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:38:19,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104422 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:38:24,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:38:24,191::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0266162 s, 154 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:38:29,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:38:29,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796722 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:38:34,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:38:34,236::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000989528 s, 4.1 MB/s\n'; <rc> = 0
>Thread-14179::WARNING::2013-03-17 12:38:38,249::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:38:38,255::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:38:39,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:38:39,402::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088451 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:38:44,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:38:44,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193086 s, 212 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:38:48,258::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:38:48,260::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:38:48,658::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:38:48,666::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:38:48,667::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:38:49,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:38:49,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00786986 s, 520 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:38:54,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:38:54,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147943 s, 277 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:38:59,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:38:59,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000691496 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:39:04,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:39:04,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161387 s, 254 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:39:09,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:39:09,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0245885 s, 167 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:39:14,445::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:39:14,478::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793466 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:39:19,697::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:39:19,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000730994 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:39:24,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:39:24,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100096 s, 409 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:39:29,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:39:29,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771893 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:39:34,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:39:34,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841629 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:39:39,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:39:39,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793938 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:39:44,576::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 12:39:44,578::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 12:39:44,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:39:44,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0112226 s, 182 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:39:44,630::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0112226 s, 182 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 12:39:44,631::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 12:39:44,631::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 12:39:44,633::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 12:39:44,634::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:39:44,635::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:39:44,635::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:39:44,636::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:39:44,636::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:39:44,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:39:45,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:39:45,082::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:39:45,083::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 12:39:45,086::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 12:39:45,087::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 12:39:45,088::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 12:39:45,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:39:45,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:39:45,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:39:45,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0287042 s, 143 kB/s\n'; <rc> = 0
>Thread-14180::WARNING::2013-03-17 12:39:48,696::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:39:48,703::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:39:49,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:39:49,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0245691 s, 167 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:39:55,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:39:55,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744658 s, 5.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:39:58,706::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:39:58,707::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:39:59,137::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:39:59,144::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:39:59,146::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:39:59,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:40:00,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00974425 s, 420 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:40:05,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:40:05,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167992 s, 244 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:40:10,092::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:40:10,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153542 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:40:15,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:40:15,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0022122 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:40:20,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:40:20,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00278386 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:40:25,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:40:25,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018993 s, 216 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:40:30,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:40:30,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0241617 s, 170 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:40:35,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:40:35,899::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129451 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:40:40,349::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 12:40:40,350::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 12:40:40,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:40:40,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000673432 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:40:40,384::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000673432 s, 3.0 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 12:40:40,385::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 12:40:40,385::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 12:40:40,387::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 12:40:40,388::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:40:40,389::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:40:40,389::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:40:40,389::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:40:40,390::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:40:40,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:40:40,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:40:40,820::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:40:40,822::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 12:40:40,824::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 12:40:40,825::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 12:40:40,826::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 12:40:40,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:40:41,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:40:41,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:40:41,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00367078 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:40:45,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:40:45,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190649 s, 215 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:40:51,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:40:51,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000655486 s, 6.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:40:56,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:40:56,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789315 s, 5.2 MB/s\n'; <rc> = 0
>Thread-14181::WARNING::2013-03-17 12:40:59,175::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:40:59,179::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:41:01,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:41:01,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924742 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:41:06,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:41:06,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100457 s, 4.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:41:09,180::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 12:41:09,182::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:41:09,183::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:41:09,586::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:41:09,593::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:41:09,594::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:41:11,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:41:11,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018879 s, 217 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:41:16,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:41:16,238::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123751 s, 331 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:41:21,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:41:21,633::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160554 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:41:26,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:41:26,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150903 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:41:31,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:41:31,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017074 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:41:36,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:41:36,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0019175 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:41:41,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:41:41,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152271 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:41:46,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:41:46,450::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0353446 s, 116 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:41:51,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:41:51,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160581 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:41:56,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:41:56,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0242476 s, 169 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:42:01,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:42:01,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145767 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:42:06,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:42:06,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080165 s, 5.1 MB/s\n'; <rc> = 0
>Thread-14182::WARNING::2013-03-17 12:42:09,623::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:42:09,626::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:42:12,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:42:12,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000974204 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:42:16,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:42:16,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00447072 s, 916 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:42:19,628::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:42:19,629::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:42:20,102::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:42:20,109::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:42:20,110::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:42:22,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:42:22,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00937463 s, 437 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:42:26,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:42:26,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900102 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:42:32,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:42:32,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846046 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:42:36,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:42:36,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131846 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:42:42,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:42:42,266::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835787 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:42:46,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:42:46,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0235777 s, 174 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:42:52,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:42:52,333::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790827 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:42:56,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:42:56,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00555391 s, 737 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:43:02,373::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:43:02,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126726 s, 323 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:43:06,922::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:43:06,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808222 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:43:12,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:43:12,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133641 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:43:16,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:43:16,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893009 s, 4.6 MB/s\n'; <rc> = 0
>Thread-14183::WARNING::2013-03-17 12:43:20,143::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:43:20,148::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:43:22,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:43:22,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810013 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:43:27,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:43:27,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897453 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:43:30,151::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:43:30,153::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:43:30,620::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:43:30,627::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:43:30,629::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:43:32,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:43:32,671::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130602 s, 314 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:43:37,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:43:37,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121363 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:43:42,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:43:42,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147414 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:43:47,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:43:47,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728867 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:43:52,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:43:52,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0244375 s, 168 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:43:57,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:43:57,297::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000949929 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:44:02,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:44:02,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777289 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:44:07,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:44:07,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192536 s, 213 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:44:12,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:44:12,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000954906 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:44:17,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:44:17,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0209061 s, 196 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:44:23,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:44:23,060::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014685 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:44:27,429::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:44:27,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123533 s, 3.3 MB/s\n'; <rc> = 0
>Thread-14184::WARNING::2013-03-17 12:44:30,655::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:44:30,658::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:44:33,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:44:33,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767093 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:44:37,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:44:37,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751152 s, 5.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:44:40,673::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:44:40,674::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:44:41,086::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:44:41,092::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:44:41,094::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:44:43,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:44:43,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172486 s, 237 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:44:47,509::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 12:44:47,510::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 12:44:47,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:44:47,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0468374 s, 43.7 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:44:47,601::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0468374 s, 43.7 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 12:44:47,601::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 12:44:47,602::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 12:44:47,603::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 12:44:47,604::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:44:47,605::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:44:47,605::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:44:47,606::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:44:47,606::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:44:47,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:44:48,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:44:48,040::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:44:48,041::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 12:44:48,044::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 12:44:48,045::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 12:44:48,046::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 12:44:48,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:44:48,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:44:48,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:44:48,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013435 s, 305 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:44:53,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:44:53,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175375 s, 234 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:44:58,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:44:58,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858226 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:45:03,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:45:03,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.025003 s, 164 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:45:08,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:45:08,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136988 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:45:13,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:45:13,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139893 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:45:18,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:45:18,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177259 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:45:23,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:45:23,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158395 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:45:28,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:45:28,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104147 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:45:33,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:45:33,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744538 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:45:38,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:45:38,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00400849 s, 1.0 MB/s\n'; <rc> = 0
>Thread-14185::WARNING::2013-03-17 12:45:41,119::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:45:41,126::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:45:43,654::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 12:45:43,655::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 12:45:43,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:45:43,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00140642 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:45:43,701::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00140642 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 12:45:43,701::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 12:45:43,701::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 12:45:43,703::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 12:45:43,704::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:45:43,705::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:45:43,705::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:45:43,706::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:45:43,706::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:45:43,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:45:44,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:45:44,177::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:45:44,178::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 12:45:44,181::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 12:45:44,182::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 12:45:44,182::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 12:45:44,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:45:44,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:45:44,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:45:44,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101493 s, 404 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:45:48,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:45:48,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809324 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:45:51,128::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:45:51,130::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:45:51,554::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:45:51,561::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:45:51,562::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:45:54,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:45:54,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00642153 s, 638 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:45:58,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:45:58,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000977165 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:46:04,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:46:04,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815687 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:46:08,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:46:08,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138568 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:46:14,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:46:14,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00289838 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:46:18,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:46:18,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00188192 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:46:24,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:46:24,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000905857 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:46:28,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:46:28,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0046907 s, 873 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:46:34,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:46:34,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151279 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:46:38,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:46:38,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000722413 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:46:44,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:46:45,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0252458 s, 162 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:46:48,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:46:48,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000962911 s, 4.3 MB/s\n'; <rc> = 0
>Thread-14186::WARNING::2013-03-17 12:46:51,588::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:46:51,592::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:46:55,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:46:55,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100331 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:46:58,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:46:58,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100462 s, 4.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:47:01,594::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 12:47:01,595::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:47:01,596::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:47:02,078::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:47:02,085::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:47:02,086::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:47:05,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:47:05,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0195482 s, 210 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:47:09,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:47:09,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000719754 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:47:15,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:47:15,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137298 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:47:19,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:47:19,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136857 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:47:25,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:47:25,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0268892 s, 152 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:47:29,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:47:29,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751539 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:47:35,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:47:35,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144695 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:47:39,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:47:39,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134951 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:47:45,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:47:45,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000986408 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:47:49,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:47:49,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858726 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:47:55,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:47:55,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159231 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:47:59,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:47:59,236::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147827 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14187::WARNING::2013-03-17 12:48:02,125::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:48:02,131::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:48:05,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:48:05,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156004 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:48:09,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:48:09,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861395 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:48:12,134::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:48:12,135::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:48:12,605::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:48:12,611::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:48:12,612::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:48:15,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:48:15,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203851 s, 201 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:48:19,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:48:19,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808589 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:48:25,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:48:25,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082469 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:48:29,312::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:48:29,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787628 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:48:35,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:48:35,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841019 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:48:39,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:48:39,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842408 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:48:45,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:48:45,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147556 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:48:49,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:48:49,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00393218 s, 1.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:48:56,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:48:56,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884513 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:48:59,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:48:59,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136466 s, 300 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:49:06,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:49:06,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146021 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:49:09,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:49:09,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149464 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14188::WARNING::2013-03-17 12:49:12,638::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:49:12,647::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:49:16,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:49:16,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000944945 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:49:19,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:49:19,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793903 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:49:22,659::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:49:22,660::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:49:23,054::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:49:23,061::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:49:23,062::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:49:26,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:49:26,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188559 s, 217 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:49:29,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:49:29,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152366 s, 269 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:49:36,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:49:36,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773878 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:49:39,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:49:39,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124686 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:49:46,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:49:46,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102695 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:49:49,725::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 12:49:49,726::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 12:49:49,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:49:49,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000816827 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:49:49,756::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000816827 s, 2.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 12:49:49,757::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 12:49:49,757::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 12:49:49,758::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 12:49:49,760::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:49:49,760::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:49:49,761::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:49:49,761::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:49:49,761::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:49:49,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:49:50,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:49:50,192::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:49:50,193::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 12:49:50,195::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 12:49:50,196::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 12:49:50,197::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 12:49:50,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:49:50,593::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:49:50,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:49:50,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916549 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:49:56,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:49:56,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159867 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:50:00,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:50:00,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822576 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:50:06,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:50:06,622::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176676 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:50:10,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:50:10,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167773 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:50:16,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:50:16,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0009067 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:50:20,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:50:20,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000969357 s, 4.2 MB/s\n'; <rc> = 0
>Thread-14189::WARNING::2013-03-17 12:50:23,088::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:50:23,090::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:50:26,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:50:26,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0251801 s, 163 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:50:30,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:50:30,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919952 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:50:33,100::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:50:33,101::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:50:33,584::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:50:33,591::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:50:33,592::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:50:36,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:50:36,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112988 s, 363 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:50:40,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:50:40,819::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160399 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:50:46,914::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 12:50:46,915::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 12:50:46,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:50:46,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00144009 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:50:46,948::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00144009 s, 1.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 12:50:46,949::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 12:50:46,949::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 12:50:46,951::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 12:50:46,953::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:50:46,954::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:50:46,954::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:50:46,955::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:50:46,955::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:50:46,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:50:47,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:50:47,389::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:50:47,390::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 12:50:47,392::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 12:50:47,394::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 12:50:47,394::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 12:50:47,396::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:50:47,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:50:47,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:50:47,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153525 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:50:50,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:50:50,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893924 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:50:57,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:50:57,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0259464 s, 158 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:51:00,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:51:00,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802619 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:51:07,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:51:07,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000723137 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:51:10,899::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:51:10,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132762 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:51:17,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:51:18,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138899 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:51:20,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:51:20,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141496 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:51:28,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:51:28,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160861 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:51:30,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:51:31,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114211 s, 3.6 MB/s\n'; <rc> = 0
>Thread-14190::WARNING::2013-03-17 12:51:33,615::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:51:33,619::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:51:38,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:51:38,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833612 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:51:41,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:51:41,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831052 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:51:43,621::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:51:43,622::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:51:44,146::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:51:44,152::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:51:44,154::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:51:48,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:51:48,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176231 s, 232 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:51:51,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:51:51,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101424 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:51:58,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:51:58,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799084 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:52:01,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:52:01,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000946564 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:52:08,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:52:08,448::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147181 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:52:11,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:52:11,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140206 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:52:18,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:52:18,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000939168 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:52:21,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:52:21,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146195 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:52:28,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:52:28,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911144 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:52:31,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:52:31,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165916 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:52:38,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:52:38,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167397 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:52:41,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:52:41,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883147 s, 4.6 MB/s\n'; <rc> = 0
>Thread-14191::WARNING::2013-03-17 12:52:44,179::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:52:44,182::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:52:48,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:52:48,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892438 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:52:51,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:52:51,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756665 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:52:54,191::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 12:52:54,192::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:52:54,193::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:52:54,698::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:52:54,704::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:52:54,706::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:52:58,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:52:58,853::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0226819 s, 181 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:53:01,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:53:01,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000931025 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:53:08,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:53:08,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091858 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:53:11,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:53:11,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866101 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:53:18,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:53:19,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143622 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:53:21,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:53:21,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125632 s, 326 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:53:29,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:53:29,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884522 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:53:31,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:53:31,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167943 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:53:39,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:53:39,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148678 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:53:41,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:53:41,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00763149 s, 537 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:53:49,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:53:49,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132927 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:53:51,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:53:51,633::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00253071 s, 1.6 MB/s\n'; <rc> = 0
>Thread-14192::WARNING::2013-03-17 12:53:54,735::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:53:54,739::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:53:59,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:53:59,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00682739 s, 600 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:54:01,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:54:01,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00190909 s, 2.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:54:04,741::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:54:04,742::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:54:05,273::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:54:05,280::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:54:05,281::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:54:09,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:54:09,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0227523 s, 180 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:54:11,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:54:11,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159353 s, 257 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:54:19,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:54:19,494::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107816 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:54:21,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:54:21,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146826 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:54:29,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:54:29,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166945 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:54:31,857::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:54:31,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901458 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:54:39,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:54:39,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780702 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:54:41,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:54:41,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199073 s, 206 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:54:49,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:54:49,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163415 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:54:51,945::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 12:54:51,947::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 12:54:51,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:54:51,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000741928 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:54:51,977::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000741928 s, 2.8 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 12:54:51,978::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 12:54:51,978::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 12:54:51,980::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 12:54:51,981::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:54:51,981::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:54:51,982::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:54:51,982::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:54:51,983::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:54:51,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:54:52,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:54:52,409::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:54:52,410::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 12:54:52,412::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 12:54:52,414::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 12:54:52,414::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 12:54:52,415::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:54:52,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:54:52,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:54:52,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142017 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:54:59,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:54:59,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144564 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:55:02,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:55:02,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140207 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14193::WARNING::2013-03-17 12:55:05,309::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:55:05,313::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:55:09,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:55:09,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826304 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:55:12,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:55:12,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787621 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:55:15,316::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:55:15,317::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:55:15,805::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:55:15,812::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:55:15,813::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:55:19,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:55:19,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.025063 s, 163 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:55:22,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:55:22,992::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758888 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:55:30,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:55:30,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137074 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:55:32,997::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:55:33,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013453 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:55:40,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:55:40,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904365 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:55:43,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:55:43,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855117 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:55:50,177::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 12:55:50,178::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 12:55:50,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:55:50,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000891106 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:55:50,218::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000891106 s, 2.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 12:55:50,219::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 12:55:50,219::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 12:55:50,220::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 12:55:50,221::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:55:50,222::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:55:50,222::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:55:50,223::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:55:50,223::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 12:55:50,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:55:50,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:55:50,656::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:55:50,657::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 12:55:50,660::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 12:55:50,661::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 12:55:50,661::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 12:55:50,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:55:51,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:55:51,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:55:51,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00638279 s, 642 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:55:53,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:55:53,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075371 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:56:01,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:56:01,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144513 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:56:03,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:56:03,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00226749 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:56:11,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:56:11,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108091 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:56:13,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:56:13,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178317 s, 2.3 MB/s\n'; <rc> = 0
>Thread-14194::WARNING::2013-03-17 12:56:15,852::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:56:15,858::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:56:21,280::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:56:21,309::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819182 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:56:23,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:56:23,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800701 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:56:25,863::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:56:25,864::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:56:26,332::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:56:26,339::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:56:26,340::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:56:31,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:56:31,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00589751 s, 695 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:56:33,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:56:33,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792761 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:56:41,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:56:41,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00926668 s, 442 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:56:43,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:56:43,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00191704 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:56:51,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:56:51,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884632 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:56:53,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:56:53,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792271 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:57:01,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:57:01,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0248461 s, 165 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:57:03,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:57:03,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143537 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:57:11,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:57:11,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166857 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:57:13,395::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:57:13,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118298 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:57:21,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:57:21,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131668 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:57:23,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:57:23,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012817 s, 3.2 MB/s\n'; <rc> = 0
>Thread-14195::WARNING::2013-03-17 12:57:26,367::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:57:26,370::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:57:31,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:57:31,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000981634 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:57:33,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:57:33,521::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814931 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:57:36,373::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:57:36,374::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:57:36,776::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:57:36,782::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:57:36,783::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:57:41,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:57:41,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078357 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:57:43,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:57:43,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800225 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:57:51,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:57:52,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847737 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:57:53,560::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:57:53,602::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147843 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:58:02,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:58:02,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158674 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:58:03,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:58:03,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144592 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:58:12,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:58:12,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752143 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:58:13,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:58:13,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160886 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:58:22,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:58:22,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924621 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:58:23,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:58:23,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145586 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:58:32,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:58:32,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014934 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:58:33,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:58:33,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771628 s, 5.3 MB/s\n'; <rc> = 0
>Thread-14196::WARNING::2013-03-17 12:58:36,808::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:58:36,812::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:58:42,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:58:42,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747693 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:58:43,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:58:43,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161777 s, 2.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:58:46,814::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 12:58:46,815::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:58:46,816::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:58:47,271::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:58:47,278::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:58:47,280::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 12:58:52,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:58:52,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00244824 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:58:53,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:58:53,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806373 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:59:02,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:59:02,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085993 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:59:03,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:59:04,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107243 s, 382 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:59:12,571::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:59:12,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0280385 s, 146 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:59:14,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:59:14,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782136 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:59:22,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:59:22,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0245867 s, 167 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:59:24,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:59:24,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011246 s, 364 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:59:32,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:59:32,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:59:34,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:59:34,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155791 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 12:59:42,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:59:42,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154943 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:59:44,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:59:44,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145624 s, 281 kB/s\n'; <rc> = 0
>Thread-14197::WARNING::2013-03-17 12:59:47,316::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 12:59:47,321::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 12:59:52,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 12:59:53,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813741 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:59:54,243::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 12:59:54,249::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 12:59:54,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:59:54,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00182161 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:59:54,280::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00182161 s, 1.1 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 12:59:54,281::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 12:59:54,281::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 12:59:54,283::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 12:59:54,284::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:59:54,284::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:59:54,285::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:59:54,285::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:59:54,286::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 12:59:54,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:59:54,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:59:54,705::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 12:59:54,706::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 12:59:54,709::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 12:59:54,710::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 12:59:54,710::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 12:59:54,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:59:55,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 12:59:55,060::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 12:59:55,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767865 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 12:59:57,324::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 12:59:57,326::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 12:59:57,724::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 12:59:57,730::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 12:59:57,732::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:00:03,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:00:03,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867896 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:00:05,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:00:05,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165626 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:00:13,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:00:13,175::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015228 s, 269 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:00:15,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:00:15,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148299 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:00:23,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:00:23,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796287 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:00:25,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:00:25,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145506 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:00:33,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:00:33,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00090822 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:00:35,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:00:35,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849747 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:00:43,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:00:43,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151477 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:00:45,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:00:45,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743595 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:00:53,435::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 13:00:53,440::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 13:00:53,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:00:53,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0281399 s, 72.8 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:00:53,509::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0281399 s, 72.8 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 13:00:53,509::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 13:00:53,510::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 13:00:53,511::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 13:00:53,512::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:00:53,515::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:00:53,515::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:00:53,515::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:00:53,516::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:00:53,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:00:53,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:00:53,930::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:00:53,931::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 13:00:53,934::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 13:00:53,935::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 13:00:53,936::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 13:00:53,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:00:54,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:00:54,305::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:00:54,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111608 s, 367 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:00:55,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:00:55,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199823 s, 205 kB/s\n'; <rc> = 0
>Thread-14198::WARNING::2013-03-17 13:00:57,761::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:00:57,767::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:01:04,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:01:04,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0261172 s, 157 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:01:05,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:01:05,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000961918 s, 4.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:01:07,771::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:01:07,772::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:01:08,188::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:01:08,194::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:01:08,196::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:01:14,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:01:14,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778253 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:01:15,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:01:15,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00215492 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:01:24,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:01:24,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126084 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:01:25,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:01:25,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0036268 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:01:34,648::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:01:34,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141524 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:01:35,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:01:35,513::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000710283 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:01:44,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:01:44,793::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820026 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:01:45,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:01:45,561::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146196 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:01:54,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:01:54,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118927 s, 344 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:01:55,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:01:55,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00242049 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:02:04,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:02:04,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000725605 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:02:05,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:02:05,658::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856044 s, 4.8 MB/s\n'; <rc> = 0
>Thread-14199::WARNING::2013-03-17 13:02:08,220::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:02:08,224::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:02:14,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:02:15,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00182972 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:02:15,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:02:15,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00217916 s, 1.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:02:18,227::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:02:18,232::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:02:18,654::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:02:18,661::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:02:18,662::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:02:25,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:02:25,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0267434 s, 153 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:02:25,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:02:25,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834756 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:02:35,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:02:35,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823899 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:02:35,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:02:35,766::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100775 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:02:45,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:02:45,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139107 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:02:45,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:02:45,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000903304 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:02:55,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:02:55,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139249 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:02:55,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:02:55,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00097013 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:03:05,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:03:05,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108039 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:03:05,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:03:05,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130684 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:03:15,455::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:03:15,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0278249 s, 147 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:03:15,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:03:15,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089254 s, 4.6 MB/s\n'; <rc> = 0
>Thread-14200::WARNING::2013-03-17 13:03:18,700::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:03:18,704::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:03:25,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:03:25,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859538 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:03:25,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:03:25,963::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828669 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:03:28,706::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:03:28,708::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:03:29,122::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:03:29,129::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:03:29,131::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:03:35,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:03:35,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0252901 s, 162 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:03:35,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:03:36,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107259 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:03:45,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:03:45,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0243423 s, 168 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:03:46,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:03:46,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888183 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:03:55,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:03:55,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127771 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:03:56,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:03:56,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00668264 s, 613 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:04:05,899::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:04:05,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00237918 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:04:06,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:04:06,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147354 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:04:15,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:04:16,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169077 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:04:16,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:04:16,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138693 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:04:26,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:04:26,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125503 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:04:26,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:04:26,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159589 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14201::WARNING::2013-03-17 13:04:29,155::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:04:29,159::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:04:36,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:04:36,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150405 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:04:36,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:04:36,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010047 s, 4.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:04:39,160::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 13:04:39,161::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:04:39,162::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:04:39,577::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:04:39,585::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:04:39,586::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:04:46,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:04:46,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000683797 s, 6.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:04:46,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:04:46,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166642 s, 246 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:04:56,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:04:56,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148028 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:04:56,419::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 13:04:56,421::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 13:04:56,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:04:56,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000872818 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:04:56,447::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000872818 s, 2.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 13:04:56,448::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 13:04:56,448::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 13:04:56,450::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 13:04:56,451::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:04:56,452::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:04:56,452::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:04:56,453::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:04:56,453::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:04:56,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:04:56,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:04:56,871::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:04:56,873::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 13:04:56,875::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 13:04:56,876::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 13:04:56,877::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 13:04:56,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:04:57,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:04:57,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:04:57,366::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102566 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:05:06,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:05:06,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759251 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:05:07,373::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:05:07,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177793 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:05:16,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:05:16,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103978 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:05:17,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:05:17,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000941129 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:05:26,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:05:26,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870293 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:05:27,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:05:27,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151823 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:05:36,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:05:36,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092632 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:05:37,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:05:37,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159326 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14202::WARNING::2013-03-17 13:05:39,614::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:05:39,618::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:05:46,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:05:46,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806905 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:05:47,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:05:47,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078098 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:05:49,620::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:05:49,621::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:05:50,079::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:05:50,086::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:05:50,087::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:05:56,780::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 13:05:56,783::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 13:05:56,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:05:56,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000890521 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:05:56,819::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000890521 s, 2.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 13:05:56,819::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 13:05:56,820::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 13:05:56,822::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 13:05:56,823::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:05:56,824::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:05:56,824::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:05:56,824::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:05:56,825::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:05:56,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:05:57,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:05:57,247::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:05:57,248::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 13:05:57,250::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 13:05:57,252::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 13:05:57,252::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 13:05:57,254::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:05:57,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:05:57,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152305 s, 269 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:05:57,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:05:57,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:05:57,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162748 s, 252 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:06:07,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:06:07,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00095965 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:06:07,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:06:07,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793073 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:06:17,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:06:17,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160358 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:06:17,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:06:17,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855781 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:06:27,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:06:27,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157793 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:06:27,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:06:27,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770858 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:06:37,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:06:37,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136184 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:06:38,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:06:38,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811124 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:06:47,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:06:47,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137606 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:06:48,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:06:48,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155217 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14203::WARNING::2013-03-17 13:06:50,116::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:06:50,120::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 13:06:57,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:06:57,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149149 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:06:58,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:06:58,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0265147 s, 154 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:07:00,130::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:07:00,132::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:07:00,603::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:07:00,610::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:07:00,612::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:07:07,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:07:07,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158752 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:07:08,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:07:08,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159378 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:07:17,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:07:17,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853036 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:07:18,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:07:18,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000929354 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:07:28,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:07:28,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824039 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:07:28,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:07:28,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00254785 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:07:38,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:07:38,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147396 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:07:38,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:07:38,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189021 s, 217 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:07:48,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:07:48,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161413 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:07:48,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:07:48,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0253896 s, 161 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:07:58,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:07:58,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772076 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:07:58,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:07:58,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875055 s, 4.7 MB/s\n'; <rc> = 0
>Thread-14204::WARNING::2013-03-17 13:08:00,636::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:08:00,640::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 13:08:08,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:08:08,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00283173 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:08:08,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:08:08,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000719923 s, 5.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:08:10,645::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:08:10,646::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:08:11,099::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:08:11,106::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:08:11,107::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:08:18,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:08:18,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00073185 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:08:18,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:08:18,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142033 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:08:28,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:08:28,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158952 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:08:29,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:08:29,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789995 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:08:38,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:08:38,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001271 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:08:39,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:08:39,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829952 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:08:48,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:08:48,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141168 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:08:49,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:08:49,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0265382 s, 154 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:08:58,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:08:58,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00328042 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:08:59,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:08:59,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0071791 s, 571 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:09:08,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:09:08,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019602 s, 209 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:09:09,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:09:09,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102418 s, 4.0 MB/s\n'; <rc> = 0
>Thread-14205::WARNING::2013-03-17 13:09:11,130::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:09:11,134::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 13:09:18,596::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:09:18,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105511 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:09:19,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:09:19,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839487 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:09:21,168::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:09:21,170::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:09:21,562::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:09:21,569::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:09:21,570::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:09:28,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:09:28,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150237 s, 273 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:09:29,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:09:29,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880752 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:09:38,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:09:38,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143457 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:09:39,612::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:09:39,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160796 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:09:48,741::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:09:48,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768053 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:09:49,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:09:49,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015882 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:09:58,775::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 13:09:58,777::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 13:09:58,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:09:58,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00100325 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:09:58,809::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00100325 s, 2.0 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 13:09:58,809::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 13:09:58,809::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 13:09:58,811::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 13:09:58,812::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:09:58,813::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:09:58,813::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:09:58,813::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:09:58,814::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:09:58,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:09:59,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:09:59,256::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:09:59,257::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 13:09:59,260::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 13:09:59,261::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 13:09:59,262::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 13:09:59,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:09:59,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:09:59,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:09:59,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0028079 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:09:59,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:09:59,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135572 s, 302 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:10:09,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:10:09,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142056 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:10:09,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:10:09,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085805 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:10:19,796::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:10:19,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011466 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:10:19,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:10:19,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00188659 s, 2.2 MB/s\n'; <rc> = 0
>Thread-14206::WARNING::2013-03-17 13:10:21,606::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:10:21,610::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 13:10:29,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:10:29,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000914974 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:10:30,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:10:30,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849962 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:10:31,623::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 13:10:31,624::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:10:31,625::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:10:32,067::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:10:32,073::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:10:32,075::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:10:39,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:10:39,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806492 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:10:40,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:10:40,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821996 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:10:49,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:10:49,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134506 s, 305 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:10:50,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:10:50,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862167 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:10:59,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:11:00,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00306135 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:11:00,216::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 13:11:00,217::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 13:11:00,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:11:00,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00101762 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:11:00,250::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00101762 s, 2.0 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 13:11:00,250::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 13:11:00,250::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 13:11:00,252::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 13:11:00,253::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:11:00,254::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:11:00,254::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:11:00,254::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:11:00,255::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:11:00,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:11:00,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:11:00,678::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:11:00,679::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 13:11:00,682::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 13:11:00,683::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 13:11:00,683::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 13:11:00,685::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:11:01,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:11:01,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:11:01,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158609 s, 258 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:11:10,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:11:10,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000726148 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:11:11,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:11:11,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0233608 s, 175 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:11:20,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:11:20,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749701 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:11:21,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:11:21,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152072 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:11:30,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:11:30,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00259776 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:11:31,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:11:31,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0428388 s, 95.6 kB/s\n'; <rc> = 0
>Thread-14207::WARNING::2013-03-17 13:11:32,103::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:11:32,109::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 13:11:40,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:11:40,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151448 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:11:41,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:11:41,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119704 s, 342 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:11:42,113::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:11:42,114::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:11:42,557::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:11:42,564::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:11:42,565::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:11:50,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:11:50,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780365 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:11:51,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:11:51,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.025953 s, 158 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:12:00,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:12:00,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754197 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:12:01,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:12:02,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794557 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:12:10,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:12:10,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907956 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:12:12,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:12:12,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087557 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:12:20,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:12:20,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00073991 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:12:22,129::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:12:22,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136421 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:12:30,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:12:30,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142862 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:12:32,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:12:32,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823113 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:12:40,395::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:12:40,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169043 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:12:42,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:12:42,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00539808 s, 759 kB/s\n'; <rc> = 0
>Thread-14208::WARNING::2013-03-17 13:12:42,595::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:12:42,599::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 13:12:50,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:12:50,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774528 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:12:52,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:12:52,653::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:12:52,656::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:12:52,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0280875 s, 146 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:12:53,102::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:12:53,107::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:12:53,109::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:13:00,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:13:00,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887712 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:13:02,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:13:02,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0242902 s, 169 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:13:10,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:13:10,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00270178 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:13:12,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:13:12,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0250243 s, 164 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:13:20,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:13:20,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169817 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:13:22,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:13:22,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000942602 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:13:30,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:13:30,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821354 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:13:33,019::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:13:33,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749185 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:13:40,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:13:40,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161779 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:13:43,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:13:43,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755394 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:13:50,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:13:50,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104106 s, 3.9 MB/s\n'; <rc> = 0
>Thread-14209::WARNING::2013-03-17 13:13:53,135::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:13:53,139::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:13:53,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:13:53,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101153 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:14:00,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:14:00,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015028 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:14:03,141::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:14:03,143::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:14:03,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:14:03,287::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00818594 s, 500 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:14:03,593::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:14:03,598::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:14:03,600::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:14:10,796::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:14:10,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0244262 s, 168 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:14:13,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:14:13,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163579 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:14:20,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:14:20,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140845 s, 291 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:14:23,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:14:23,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00379034 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:14:30,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:14:30,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821358 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:14:33,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:14:33,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160269 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:14:40,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:14:40,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134088 s, 305 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:14:43,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:14:43,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762714 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:14:51,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:14:51,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164311 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:14:53,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:14:53,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133909 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:15:01,049::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 13:15:01,051::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 13:15:01,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:15:01,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0189236 s, 108 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:15:01,114::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0189236 s, 108 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 13:15:01,114::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 13:15:01,115::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 13:15:01,124::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 13:15:01,126::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:15:01,129::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:15:01,130::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:15:01,131::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:15:01,131::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:15:01,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:15:01,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:15:01,574::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:15:01,575::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 13:15:01,578::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 13:15:01,579::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 13:15:01,580::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 13:15:01,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:15:01,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:15:01,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:15:01,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165755 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14210::WARNING::2013-03-17 13:15:03,626::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:15:03,629::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:15:03,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:15:03,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128632 s, 318 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:15:12,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:15:12,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0018098 s, 2.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:15:13,632::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:15:13,633::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:15:13,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:15:13,866::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116831 s, 351 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:15:14,095::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:15:14,101::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:15:14,102::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:15:22,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:15:22,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862168 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:15:23,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:15:23,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151782 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:15:32,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:15:32,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813585 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:15:34,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:15:34,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823539 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:15:42,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:15:42,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785654 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:15:44,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:15:44,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00999256 s, 410 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:15:52,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:15:52,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771143 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:15:54,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:15:54,191::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879649 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:16:02,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:16:02,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147576 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:16:04,234::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 13:16:04,236::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 13:16:04,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:16:04,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00563145 s, 364 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:16:04,279::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00563145 s, 364 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 13:16:04,279::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 13:16:04,280::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 13:16:04,281::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 13:16:04,282::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:16:04,284::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:16:04,284::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:16:04,285::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:16:04,285::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:16:04,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:16:04,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:16:04,719::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:16:04,720::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 13:16:04,722::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 13:16:04,723::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 13:16:04,724::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 13:16:04,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:16:05,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:16:05,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:16:05,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00690844 s, 593 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:16:12,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:16:12,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753754 s, 5.4 MB/s\n'; <rc> = 0
>Thread-14211::WARNING::2013-03-17 13:16:14,128::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:16:14,135::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:16:15,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:16:15,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127536 s, 321 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:16:22,292::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:16:22,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000731004 s, 5.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:16:24,141::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 13:16:24,142::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:16:24,144::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:16:24,566::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:16:24,572::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:16:24,574::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:16:25,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:16:25,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133334 s, 307 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:16:32,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:16:32,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100962 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:16:35,405::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:16:35,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000694773 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:16:42,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:16:42,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816536 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:16:45,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:16:45,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087992 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:16:52,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:16:52,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000939699 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:16:55,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:16:55,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853211 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:17:02,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:17:02,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143128 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:17:05,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:17:05,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000691728 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:17:12,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:17:12,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861664 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:17:15,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:17:15,733::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133811 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:17:22,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:17:22,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133298 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14212::WARNING::2013-03-17 13:17:24,610::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:17:24,617::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:17:25,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:17:25,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0267315 s, 153 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:17:32,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:17:32,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893487 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:17:34,620::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:17:34,621::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:17:35,074::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:17:35,080::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:17:35,082::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:17:35,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:17:35,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0198045 s, 207 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:17:42,592::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:17:42,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135113 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:17:45,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:17:45,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806957 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:17:52,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:17:52,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140717 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:17:56,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:17:56,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792877 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:18:02,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:18:02,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816146 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:18:06,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:18:06,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780183 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:18:12,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:18:12,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899812 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:18:16,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:18:16,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913985 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:18:22,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:18:22,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746502 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:18:26,230::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:18:26,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124837 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:18:32,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:18:32,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131451 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14213::WARNING::2013-03-17 13:18:35,107::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:18:35,114::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:18:36,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:18:36,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821399 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:18:42,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:18:42,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853743 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:18:45,117::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:18:45,118::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:18:45,590::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:18:45,597::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:18:45,598::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:18:46,396::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:18:46,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0208663 s, 196 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:18:52,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:18:52,912::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124753 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:18:56,490::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:18:56,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0270349 s, 152 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:19:02,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:19:03,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145286 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:19:06,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:19:06,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178533 s, 229 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:19:13,033::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:19:13,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901038 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:19:16,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:19:16,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156141 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:19:23,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:19:23,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774232 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:19:26,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:19:26,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759368 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:19:33,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:19:33,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809793 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:19:36,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:19:36,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142616 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:19:43,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:19:43,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102429 s, 4.0 MB/s\n'; <rc> = 0
>Thread-14214::WARNING::2013-03-17 13:19:45,623::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:19:45,627::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:19:46,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:19:46,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000937243 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:19:53,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:19:53,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000957376 s, 4.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:19:55,629::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:19:55,631::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:19:56,052::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:19:56,058::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:19:56,060::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:19:56,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:19:57,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0216329 s, 189 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:20:03,222::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 13:20:03,224::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 13:20:03,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:20:03,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0177896 s, 115 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:20:03,271::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0177896 s, 115 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 13:20:03,272::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 13:20:03,272::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 13:20:03,276::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 13:20:03,277::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:20:03,278::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:20:03,278::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:20:03,279::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:20:03,279::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:20:03,280::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:20:03,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:20:03,702::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:20:03,703::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 13:20:03,705::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 13:20:03,706::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 13:20:03,707::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 13:20:03,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:20:04,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:20:04,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:20:04,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177236 s, 231 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:20:07,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:20:07,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01842 s, 222 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:20:14,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:20:14,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876826 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:20:17,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:20:17,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839382 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:20:24,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:20:24,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010223 s, 401 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:20:27,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:20:27,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139256 s, 294 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:20:34,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:20:34,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082582 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:20:37,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:20:37,399::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140995 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:20:44,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:20:44,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107038 s, 383 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:20:47,439::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:20:47,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085538 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:20:54,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:20:54,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00391821 s, 1.0 MB/s\n'; <rc> = 0
>Thread-14215::WARNING::2013-03-17 13:20:56,086::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:20:56,090::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:20:57,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:20:57,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161407 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:21:04,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:21:04,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891579 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:21:06,092::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:21:06,093::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:21:06,498::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:21:06,505::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:21:06,506::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:21:07,599::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 13:21:07,602::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 13:21:07,606::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:21:07,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0088508 s, 231 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:21:07,641::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0088508 s, 231 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 13:21:07,642::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 13:21:07,642::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 13:21:07,644::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 13:21:07,645::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:21:07,646::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:21:07,646::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:21:07,647::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:21:07,647::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:21:07,648::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:21:08,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:21:08,036::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:21:08,037::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 13:21:08,039::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 13:21:08,041::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 13:21:08,041::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 13:21:08,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:21:08,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:21:08,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:21:08,439::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121509 s, 337 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:21:14,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:21:14,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809044 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:21:18,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:21:18,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143044 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:21:24,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:21:24,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765646 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:21:28,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:21:28,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0258624 s, 158 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:21:34,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:21:34,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014311 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:21:38,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:21:38,698::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163872 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:21:44,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:21:44,548::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919421 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:21:48,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:21:48,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0200173 s, 205 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:21:54,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:21:54,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778423 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:21:58,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:21:58,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145935 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:22:04,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:22:04,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120224 s, 3.4 MB/s\n'; <rc> = 0
>Thread-14216::WARNING::2013-03-17 13:22:06,538::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:22:06,544::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:22:08,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:22:08,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185667 s, 221 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:22:14,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:22:14,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075536 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:22:16,547::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 13:22:16,548::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:22:16,549::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:22:16,994::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:22:17,000::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:22:17,002::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:22:19,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:22:19,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112481 s, 364 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:22:24,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:22:24,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000918113 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:22:29,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:22:29,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753754 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:22:34,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:22:34,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162532 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:22:39,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:22:39,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00204866 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:22:44,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:22:44,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142888 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:22:49,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:22:49,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123846 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:22:54,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:22:54,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749202 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:22:59,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:22:59,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013204 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:23:04,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:23:04,907::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166182 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:23:09,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:23:09,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752746 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:23:14,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:23:14,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107696 s, 3.8 MB/s\n'; <rc> = 0
>Thread-14217::WARNING::2013-03-17 13:23:17,030::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:23:17,036::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:23:19,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:23:19,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155331 s, 264 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:23:24,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:23:25,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180874 s, 226 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:23:27,038::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:23:27,040::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:23:27,477::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:23:27,484::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:23:27,485::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:23:29,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:23:29,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185003 s, 221 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:23:35,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:23:35,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768806 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:23:39,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:23:39,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854727 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:23:45,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:23:45,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912218 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:23:49,740::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:23:49,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876446 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:23:55,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:23:55,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000910113 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:23:59,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:23:59,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000986758 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:24:05,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:24:05,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0251015 s, 163 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:24:09,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:24:09,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0098623 s, 415 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:24:15,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:24:15,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135632 s, 302 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:24:19,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:24:20,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134806 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:24:25,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:24:25,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000980768 s, 4.2 MB/s\n'; <rc> = 0
>Thread-14218::WARNING::2013-03-17 13:24:27,513::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:24:27,515::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:24:30,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:24:30,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836318 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:24:35,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:24:35,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192169 s, 213 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:24:37,518::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:24:37,519::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:24:37,958::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:24:37,965::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:24:37,967::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:24:40,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:24:40,191::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0248063 s, 165 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:24:45,482::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:24:45,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169811 s, 241 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:24:50,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:24:50,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125068 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:24:55,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:24:55,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171164 s, 239 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:25:00,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:25:00,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163332 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:25:05,595::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 13:25:05,596::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 13:25:05,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:25:05,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000821631 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:25:05,625::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000821631 s, 2.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 13:25:05,625::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 13:25:05,625::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 13:25:05,628::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 13:25:05,629::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:25:05,630::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:25:05,630::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:25:05,631::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:25:05,631::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:25:05,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:25:06,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:25:06,053::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:25:06,054::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 13:25:06,056::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 13:25:06,057::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 13:25:06,058::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 13:25:06,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:25:06,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:25:06,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:25:06,494::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00097739 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:25:10,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:25:10,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191343 s, 214 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:25:16,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:25:16,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859288 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:25:20,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:25:20,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139841 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:25:26,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:25:26,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00094303 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:25:30,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:25:30,617::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134864 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:25:36,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:25:36,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000742109 s, 5.5 MB/s\n'; <rc> = 0
>Thread-14219::WARNING::2013-03-17 13:25:38,000::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:25:38,004::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:25:40,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:25:40,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912317 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:25:46,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:25:46,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809108 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:25:48,007::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:25:48,008::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:25:48,502::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:25:48,508::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:25:48,509::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:25:50,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:25:50,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0202926 s, 202 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:25:56,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:25:56,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862039 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:26:00,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:26:00,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147229 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:26:06,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:26:06,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016029 s, 256 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:26:10,899::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 13:26:10,901::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 13:26:10,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:26:10,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000782085 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:26:10,934::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000782085 s, 2.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 13:26:10,935::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 13:26:10,935::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 13:26:10,937::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 13:26:10,938::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:26:10,939::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:26:10,939::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:26:10,939::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:26:10,940::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:26:10,941::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:26:11,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:26:11,374::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:26:11,376::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 13:26:11,381::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 13:26:11,382::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 13:26:11,383::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 13:26:11,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:26:11,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:26:11,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:26:11,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00544802 s, 752 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:26:16,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:26:16,912::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000943416 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:26:21,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:26:21,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863218 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:26:26,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:26:26,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149453 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:26:31,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:26:31,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141971 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:26:36,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:26:37,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140648 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:26:41,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:26:42,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011222 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:26:47,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:26:47,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777581 s, 5.3 MB/s\n'; <rc> = 0
>Thread-14220::WARNING::2013-03-17 13:26:48,539::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:26:48,542::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:26:52,045::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:26:52,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116087 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:26:57,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:26:57,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168011 s, 2.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:26:58,548::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:26:58,549::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:26:59,054::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:26:59,060::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:26:59,062::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:27:02,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:27:02,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0243707 s, 168 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:27:07,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:27:07,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884258 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:27:12,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:27:12,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081303 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:27:17,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:27:17,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825241 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:27:22,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:27:22,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206494 s, 198 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:27:27,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:27:27,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075924 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:27:32,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:27:32,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145323 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:27:37,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:27:37,331::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151196 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:27:42,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:27:42,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152021 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:27:47,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:27:47,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155545 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:27:52,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:27:52,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00191293 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:27:57,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:27:57,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103152 s, 4.0 MB/s\n'; <rc> = 0
>Thread-14221::WARNING::2013-03-17 13:27:59,093::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:27:59,101::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:28:02,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:28:02,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757455 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:28:07,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:28:07,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884314 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:28:09,113::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 13:28:09,114::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:28:09,115::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:28:09,559::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:28:09,566::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:28:09,567::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:28:12,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:28:12,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0452542 s, 90.5 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:28:17,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:28:17,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798327 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:28:22,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:28:22,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817948 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:28:27,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:28:27,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843744 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:28:32,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:28:32,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851431 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:28:37,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:28:37,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159187 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:28:42,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:28:43,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784305 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:28:47,592::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:28:47,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157019 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:28:53,048::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:28:53,081::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891364 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:28:57,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:28:57,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845102 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:29:03,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:29:03,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128673 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:29:07,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:29:07,738::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012519 s, 3.3 MB/s\n'; <rc> = 0
>Thread-14222::WARNING::2013-03-17 13:29:09,591::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:29:09,595::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:29:13,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:29:13,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0329369 s, 124 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:29:17,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:29:17,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884505 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:29:19,603::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:29:19,605::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:29:20,062::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:29:20,069::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:29:20,070::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:29:23,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:29:23,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0207759 s, 197 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:29:27,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:29:27,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183573 s, 223 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:29:33,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:29:33,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00671105 s, 610 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:29:37,857::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:29:37,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145392 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:29:43,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:29:43,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0220552 s, 186 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:29:47,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:29:47,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834865 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:29:53,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:29:53,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150573 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:29:57,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:29:57,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163705 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:30:03,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:30:03,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00529285 s, 774 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:30:07,977::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 13:30:07,978::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 13:30:07,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:30:08,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00129486 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:30:08,016::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00129486 s, 1.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 13:30:08,016::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 13:30:08,017::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 13:30:08,018::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 13:30:08,019::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:30:08,020::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:30:08,020::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:30:08,021::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:30:08,021::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:30:08,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:30:08,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:30:08,450::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:30:08,451::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 13:30:08,454::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 13:30:08,455::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 13:30:08,455::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 13:30:08,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:30:08,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:30:08,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:30:08,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108169 s, 379 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:30:13,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:30:13,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0289682 s, 141 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:30:18,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:30:18,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.012931 s, 317 kB/s\n'; <rc> = 0
>Thread-14223::WARNING::2013-03-17 13:30:20,102::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:30:20,106::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:30:23,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:30:23,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00197774 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:30:29,019::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:30:29,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821989 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:30:30,109::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:30:30,110::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:30:30,511::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:30:30,518::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:30:30,519::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:30:33,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:30:33,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185426 s, 221 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:30:39,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:30:39,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134446 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:30:44,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:30:44,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115183 s, 356 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:30:49,099::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:30:49,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774513 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:30:54,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:30:54,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00909851 s, 450 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:30:59,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:30:59,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743657 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:31:04,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:31:04,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0244859 s, 167 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:31:09,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:31:09,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882338 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:31:14,292::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 13:31:14,294::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 13:31:14,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:31:14,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00152518 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:31:14,338::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00152518 s, 1.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 13:31:14,338::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 13:31:14,338::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 13:31:14,340::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 13:31:14,341::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:31:14,342::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:31:14,342::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:31:14,343::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:31:14,343::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:31:14,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:31:14,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:31:14,766::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:31:14,767::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 13:31:14,770::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 13:31:14,771::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 13:31:14,771::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 13:31:14,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:31:15,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:31:15,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:31:15,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00960234 s, 427 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:31:19,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:31:19,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000930678 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:31:25,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:31:25,287::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131558 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:31:29,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:31:29,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00543339 s, 754 kB/s\n'; <rc> = 0
>Thread-14224::WARNING::2013-03-17 13:31:30,567::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:31:30,576::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:31:35,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:31:35,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860865 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:31:39,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:31:39,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00641231 s, 639 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:31:40,579::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:31:40,580::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:31:40,982::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:31:40,988::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:31:40,990::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:31:45,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:31:45,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0227241 s, 180 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:31:49,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:31:49,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128808 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:31:55,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:31:55,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135473 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:31:59,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:31:59,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079512 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:32:05,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:32:05,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130046 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:32:09,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:32:09,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00465758 s, 879 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:32:15,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:32:15,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000726189 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:32:19,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:32:19,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936431 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:32:25,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:32:25,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139263 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:32:29,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:32:29,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00219313 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:32:35,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:32:35,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146999 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:32:39,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:32:39,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0044092 s, 929 kB/s\n'; <rc> = 0
>Thread-14225::WARNING::2013-03-17 13:32:41,016::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:32:41,020::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:32:45,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:32:45,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155917 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:32:49,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:32:49,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859973 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:32:51,022::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:32:51,024::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:32:51,458::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:32:51,468::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:32:51,469::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:32:56,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:32:56,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0244476 s, 168 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:32:59,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:33:00,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778978 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:33:06,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:33:06,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748177 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:33:10,014::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:33:10,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838059 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:33:16,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:33:16,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0250834 s, 163 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:33:20,048::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:33:20,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858058 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:33:26,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:33:26,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108176 s, 379 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:33:30,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:33:30,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000733563 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:33:36,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:33:36,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916908 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:33:40,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:33:40,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00097063 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:33:46,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:33:46,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146284 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:33:50,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:33:50,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000724041 s, 5.7 MB/s\n'; <rc> = 0
>Thread-14226::WARNING::2013-03-17 13:33:51,497::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:33:51,505::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:33:56,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:33:56,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0254711 s, 161 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:34:00,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:34:00,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075417 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:34:01,513::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 13:34:01,514::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:34:01,515::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:34:02,032::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:34:02,038::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:34:02,040::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:34:06,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:34:06,667::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0360631 s, 114 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:34:10,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:34:10,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133995 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:34:16,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:34:16,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197296 s, 208 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:34:20,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:34:20,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845403 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:34:26,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:34:26,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840358 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:34:30,405::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:34:30,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743491 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:34:36,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:34:36,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0242226 s, 169 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:34:40,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:34:40,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904167 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:34:46,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:34:46,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000935886 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:34:50,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:34:50,569::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000717426 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:34:57,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:34:57,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00374319 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:35:00,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:35:00,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204847 s, 200 kB/s\n'; <rc> = 0
>Thread-14227::WARNING::2013-03-17 13:35:02,074::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:35:02,076::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:35:07,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:35:07,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904237 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:35:10,714::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 13:35:10,715::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 13:35:10,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:35:10,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000685726 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:35:10,753::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000685726 s, 3.0 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 13:35:10,753::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 13:35:10,753::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 13:35:10,755::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 13:35:10,756::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:35:10,757::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:35:10,757::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:35:10,758::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:35:10,758::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:35:10,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:35:11,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:35:11,207::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:35:11,207::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 13:35:11,210::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 13:35:11,211::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 13:35:11,212::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 13:35:11,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:35:11,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:35:11,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:35:11,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204553 s, 200 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:35:12,079::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:35:12,080::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:35:12,511::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:35:12,517::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:35:12,519::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:35:17,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:35:17,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0368857 s, 111 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:35:21,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:35:21,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108648 s, 377 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:35:27,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:35:27,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169575 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:35:31,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:35:31,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773178 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:35:37,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:35:37,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103941 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:35:41,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:35:41,843::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162155 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:35:47,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:35:47,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741979 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:35:51,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:35:51,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167562 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:35:57,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:35:57,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015017 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:36:01,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:36:02,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0207421 s, 197 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:36:07,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:36:07,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162338 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:36:12,045::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:36:12,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00531826 s, 770 kB/s\n'; <rc> = 0
>Thread-14228::WARNING::2013-03-17 13:36:12,555::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:36:12,557::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:36:17,692::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 13:36:17,693::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 13:36:17,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:36:17,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000847618 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:36:17,719::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000847618 s, 2.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 13:36:17,719::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 13:36:17,720::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 13:36:17,721::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 13:36:17,722::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:36:17,723::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:36:17,723::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:36:17,724::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:36:17,724::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:36:17,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:36:18,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:36:18,122::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:36:18,123::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 13:36:18,126::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 13:36:18,127::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 13:36:18,128::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 13:36:18,129::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:36:18,494::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:36:18,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:36:18,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00623003 s, 657 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:36:22,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:36:22,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844817 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:36:22,560::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:36:22,562::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:36:22,975::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:36:22,981::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:36:22,982::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:36:28,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:36:28,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165716 s, 247 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:36:32,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:36:32,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168216 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:36:38,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:36:38,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134542 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:36:42,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:36:42,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882454 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:36:48,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:36:48,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789303 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:36:52,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:36:52,372::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872431 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:36:58,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:36:58,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772934 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:37:02,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:37:02,424::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161766 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:37:08,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:37:08,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0246097 s, 166 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:37:12,429::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:37:12,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145093 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:37:19,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:37:19,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141482 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:37:22,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:37:22,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00312851 s, 1.3 MB/s\n'; <rc> = 0
>Thread-14229::WARNING::2013-03-17 13:37:23,011::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:37:23,014::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:37:29,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:37:29,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810595 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:37:32,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:37:32,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092829 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:37:33,016::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:37:33,018::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:37:33,428::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:37:33,435::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:37:33,436::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:37:39,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:37:39,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182845 s, 224 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:37:42,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:37:42,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0220078 s, 186 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:37:49,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:37:49,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077084 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:37:52,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:37:52,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0202052 s, 203 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:37:59,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:37:59,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858443 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:38:02,792::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:38:02,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074396 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:38:09,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:38:09,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0238822 s, 172 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:38:12,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:38:12,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0225961 s, 181 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:38:19,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:38:19,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146589 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:38:22,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:38:22,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100027 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:38:29,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:38:29,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795881 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:38:32,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:38:33,040::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118669 s, 3.5 MB/s\n'; <rc> = 0
>Thread-14230::WARNING::2013-03-17 13:38:33,460::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:38:33,464::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:38:39,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:38:39,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00541404 s, 757 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:38:43,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:38:43,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0322374 s, 127 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:38:43,467::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:38:43,468::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:38:43,947::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:38:43,954::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:38:43,955::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:38:49,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:38:49,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155486 s, 263 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:38:53,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:38:53,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140565 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:38:59,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:38:59,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140156 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:39:03,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:39:03,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138023 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:39:09,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:39:09,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812418 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:39:13,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:39:13,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804382 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:39:19,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:39:20,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759235 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:39:23,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:39:23,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783335 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:39:30,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:39:30,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138135 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:39:33,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:39:33,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105034 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:39:40,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:39:40,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00227643 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:39:43,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:39:43,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0039351 s, 1.0 MB/s\n'; <rc> = 0
>Thread-14231::WARNING::2013-03-17 13:39:43,985::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:39:43,989::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:39:50,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:39:50,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790695 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:39:53,530::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:39:53,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076098 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:39:53,991::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 13:39:53,992::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:39:53,993::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:39:54,458::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:39:54,465::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:39:54,466::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:40:00,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:40:00,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0221701 s, 185 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:40:03,605::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:40:03,667::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194868 s, 210 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:40:10,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:40:10,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885031 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:40:13,675::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 13:40:13,677::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 13:40:13,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:40:13,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0222564 s, 92.0 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:40:13,728::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0222564 s, 92.0 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 13:40:13,729::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 13:40:13,729::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 13:40:13,732::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 13:40:13,733::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:40:13,734::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:40:13,734::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:40:13,735::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:40:13,735::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:40:13,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:40:14,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:40:14,180::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:40:14,181::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 13:40:14,184::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 13:40:14,185::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 13:40:14,186::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 13:40:14,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:40:14,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:40:14,614::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:40:14,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175962 s, 233 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:40:20,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:40:20,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163244 s, 251 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:40:24,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:40:24,698::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014746 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:40:30,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:40:30,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00287274 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:40:34,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:40:34,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141163 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:40:40,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:40:40,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851961 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:40:44,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:40:44,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000967829 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:40:50,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:40:50,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802795 s, 5.1 MB/s\n'; <rc> = 0
>Thread-14232::WARNING::2013-03-17 13:40:54,490::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:40:54,495::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 13:40:54,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:40:54,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000742018 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:41:00,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:41:00,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152727 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:41:04,502::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:41:04,503::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:41:04,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:41:04,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172446 s, 238 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:41:05,020::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:41:05,025::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:41:05,027::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:41:10,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:41:10,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175714 s, 233 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:41:14,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:41:14,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156219 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:41:20,944::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 13:41:20,946::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 13:41:20,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:41:20,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00078348 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:41:20,978::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00078348 s, 2.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 13:41:20,979::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 13:41:20,979::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 13:41:20,981::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 13:41:20,982::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:41:20,982::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:41:20,983::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:41:20,983::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:41:20,983::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:41:20,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:41:21,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:41:21,419::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:41:21,420::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 13:41:21,422::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 13:41:21,423::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 13:41:21,424::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 13:41:21,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:41:21,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:41:21,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:41:21,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0223314 s, 183 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:41:24,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:41:24,992::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137533 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:41:31,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:41:31,936::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149079 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:41:34,997::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:41:35,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000918098 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:41:41,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:41:42,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140667 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:41:45,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:41:45,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00546226 s, 750 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:41:52,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:41:52,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769745 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:41:55,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:41:55,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895895 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:42:02,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:42:02,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137763 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14233::WARNING::2013-03-17 13:42:05,057::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:42:05,060::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 13:42:05,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:42:05,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00703128 s, 583 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:42:12,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:42:12,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797687 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:42:15,062::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:42:15,063::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:42:15,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:42:15,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00892803 s, 459 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:42:15,612::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:42:15,618::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:42:15,619::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:42:22,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:42:22,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0239207 s, 171 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:42:25,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:42:25,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134697 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:42:32,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:42:32,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799952 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:42:35,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:42:35,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843655 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:42:42,492::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:42:42,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000993575 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:42:45,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:42:45,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00243698 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:42:52,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:42:52,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809964 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:42:55,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:42:55,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0205899 s, 199 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:43:02,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:43:02,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157106 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:43:05,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:43:05,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0200852 s, 204 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:43:12,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:43:12,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00218112 s, 1.9 MB/s\n'; <rc> = 0
>Thread-14234::WARNING::2013-03-17 13:43:15,660::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:43:15,667::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 13:43:15,681::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:43:15,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878218 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:43:22,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:43:22,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079369 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:43:25,670::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:43:25,672::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:43:25,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:43:25,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193204 s, 212 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:43:26,144::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:43:26,149::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:43:26,151::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:43:32,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:43:32,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143185 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:43:35,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:43:35,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144233 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:43:42,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:43:42,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100016 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:43:45,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:43:45,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083952 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:43:53,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:43:53,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170945 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:43:55,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:43:55,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162989 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:44:03,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:44:03,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0250917 s, 163 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:44:05,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:44:05,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.025726 s, 159 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:44:13,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:44:13,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140511 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:44:15,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:44:16,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843616 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:44:23,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:44:23,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0222391 s, 184 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:44:26,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:44:26,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000933373 s, 4.4 MB/s\n'; <rc> = 0
>Thread-14235::WARNING::2013-03-17 13:44:26,188::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:44:26,206::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:44:33,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:44:33,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888612 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:44:36,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:44:36,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112949 s, 363 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:44:36,212::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:44:36,214::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:44:36,640::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:44:36,647::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:44:36,648::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:44:43,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:44:43,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835485 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:44:46,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:44:46,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936646 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:44:53,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:44:53,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807838 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:44:56,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:44:56,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00339409 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:45:03,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:45:03,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0263711 s, 155 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:45:06,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:45:06,287::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781869 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:45:13,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:45:13,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797947 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:45:16,291::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 13:45:16,292::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 13:45:16,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:45:16,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00123154 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:45:16,336::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00123154 s, 1.7 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 13:45:16,336::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 13:45:16,336::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 13:45:16,338::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 13:45:16,339::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:45:16,339::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:45:16,340::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:45:16,340::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:45:16,341::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:45:16,341::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:45:16,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:45:16,783::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:45:16,784::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 13:45:16,786::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 13:45:16,788::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 13:45:16,788::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 13:45:16,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:45:17,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:45:17,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:45:17,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.022849 s, 179 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:45:23,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:45:23,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148081 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:45:27,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:45:27,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143788 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:45:33,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:45:33,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134838 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14236::WARNING::2013-03-17 13:45:36,674::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:45:36,680::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 13:45:37,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:45:37,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852376 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:45:43,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:45:44,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00894553 s, 458 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:45:46,684::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 13:45:46,685::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:45:46,686::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:45:47,132::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:45:47,139::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:45:47,140::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:45:47,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:45:47,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141848 s, 289 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:45:54,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:45:54,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111606 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:45:57,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:45:57,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166673 s, 246 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:46:04,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:46:04,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0247969 s, 165 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:46:07,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:46:07,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804461 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:46:14,227::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:46:14,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132633 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:46:17,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:46:17,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879251 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:46:24,298::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 13:46:24,300::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 13:46:24,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:46:24,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00118654 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:46:24,339::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00118654 s, 1.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 13:46:24,339::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 13:46:24,339::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 13:46:24,341::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 13:46:24,342::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:46:24,343::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:46:24,343::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:46:24,343::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:46:24,344::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:46:24,345::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:46:24,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:46:24,779::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:46:24,780::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 13:46:24,783::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 13:46:24,784::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 13:46:24,784::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 13:46:24,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:46:25,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:46:25,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:46:25,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0412887 s, 99.2 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:46:27,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:46:27,513::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803714 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:46:35,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:46:35,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139934 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:46:37,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:46:37,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014624 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:46:45,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:46:45,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111647 s, 3.7 MB/s\n'; <rc> = 0
>Thread-14237::WARNING::2013-03-17 13:46:47,167::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:46:47,171::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 13:46:47,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:46:47,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867593 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:46:55,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:46:55,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158668 s, 2.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:46:57,173::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:46:57,175::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:46:57,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:46:57,651::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-22::DEBUG::2013-03-17 13:46:57,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136001 s, 301 kB/s\n'; <rc> = 0
>Thread-21::WARNING::2013-03-17 13:46:57,686::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:46:57,687::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:47:05,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:47:05,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792595 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:47:07,685::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:47:07,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925051 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:47:15,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:47:15,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00192736 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:47:17,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:47:17,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728619 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:47:25,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:47:25,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888319 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:47:27,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:47:27,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142791 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:47:35,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:47:35,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0240233 s, 171 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:47:37,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:47:37,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822995 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:47:45,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:47:45,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131743 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:47:47,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:47:47,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147782 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:47:56,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:47:56,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000937574 s, 4.4 MB/s\n'; <rc> = 0
>Thread-14238::WARNING::2013-03-17 13:47:57,715::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:47:57,718::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 13:47:57,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:47:57,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849294 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:48:06,128::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:48:06,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911864 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:48:07,721::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:48:07,722::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:48:07,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:48:07,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162255 s, 252 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:48:08,196::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:48:08,202::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:48:08,203::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:48:16,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:48:16,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920129 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:48:17,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:48:18,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00093164 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:48:26,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:48:26,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752266 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:48:28,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:48:28,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00190578 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:48:36,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:48:36,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092512 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:48:38,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:48:38,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076989 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:48:46,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:48:46,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012829 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:48:48,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:48:48,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138947 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:48:56,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:48:56,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775943 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:48:58,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:48:58,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879095 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:49:06,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:49:06,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152947 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:49:08,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-14239::WARNING::2013-03-17 13:49:08,230::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:49:08,257::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 13:49:08,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108983 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:49:16,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:49:16,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197047 s, 208 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:49:18,261::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:49:18,263::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:49:18,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:49:18,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000963364 s, 4.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:49:18,691::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:49:18,697::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:49:18,698::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:49:27,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:49:27,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0238842 s, 171 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:49:28,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:49:28,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790686 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:49:37,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:49:37,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816915 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:49:38,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:49:38,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0209377 s, 196 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:49:47,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:49:47,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143401 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:49:48,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:49:48,483::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00503679 s, 813 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:49:57,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:49:57,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814188 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:49:58,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:49:58,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172259 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:50:07,372::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:50:07,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153673 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:50:08,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:50:08,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855016 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:50:17,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:50:17,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000668062 s, 6.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:50:18,585::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 13:50:18,591::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 13:50:18,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:50:18,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0206421 s, 99.2 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:50:18,664::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0206421 s, 99.2 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 13:50:18,666::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 13:50:18,668::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 13:50:18,673::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 13:50:18,679::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:50:18,683::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:50:18,686::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:50:18,688::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:50:18,690::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:50:18,697::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-14240::WARNING::2013-03-17 13:50:18,735::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:50:18,744::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 13:50:19,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:50:19,156::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:50:19,156::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 13:50:19,159::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 13:50:19,160::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 13:50:19,160::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 13:50:19,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:50:19,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:50:19,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:50:19,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182494 s, 224 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:50:27,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:50:27,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174133 s, 2.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:50:28,748::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:50:28,749::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:50:29,178::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:50:29,185::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:50:29,186::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:50:29,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:50:29,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129895 s, 315 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:50:37,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:50:37,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797326 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:50:39,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:50:39,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000975229 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:50:48,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:50:48,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852909 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:50:49,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:50:49,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803069 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:50:58,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:50:58,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0264563 s, 155 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:50:59,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:50:59,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782145 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:51:08,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:51:08,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0224372 s, 183 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:51:09,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:51:09,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074317 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:51:18,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:51:18,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147198 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:51:19,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:51:19,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140205 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:51:28,533::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 13:51:28,534::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 13:51:28,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:51:28,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000837722 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:51:28,590::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000837722 s, 2.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 13:51:28,593::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 13:51:28,594::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 13:51:28,598::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 13:51:28,603::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:51:28,607::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:51:28,609::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:51:28,612::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:51:28,614::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:51:28,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:51:29,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:51:29,074::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:51:29,081::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 13:51:29,093::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 13:51:29,096::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 13:51:29,098::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 13:51:29,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-14241::WARNING::2013-03-17 13:51:29,215::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:51:29,223::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:51:29,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:51:29,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:51:29,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145951 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:51:29,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:51:29,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00459586 s, 891 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:51:39,226::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 13:51:39,227::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:51:39,229::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:51:39,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:51:39,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105601 s, 388 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:51:39,681::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:51:39,687::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:51:39,688::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:51:39,857::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:51:39,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183148 s, 224 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:51:49,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:51:49,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.023475 s, 174 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:51:49,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:51:49,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778407 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:51:59,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:51:59,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135053 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:51:59,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:51:59,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000699162 s, 5.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:52:09,792::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:52:09,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150224 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:52:09,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:52:10,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818353 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:52:19,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:52:19,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081177 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:52:20,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:52:20,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140986 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:52:29,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:52:30,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163368 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:52:30,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:52:30,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813106 s, 5.0 MB/s\n'; <rc> = 0
>Thread-14242::WARNING::2013-03-17 13:52:39,717::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:52:39,721::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:52:40,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:52:40,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901374 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:52:40,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:52:40,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00208059 s, 2.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:52:49,723::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:52:49,724::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:52:50,145::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:52:50,151::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:52:50,153::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:52:50,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:52:50,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:52:50,270::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0207729 s, 197 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:52:50,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128562 s, 319 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:53:00,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:53:00,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141439 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:53:00,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:53:00,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143639 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:53:10,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:53:10,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861252 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:53:10,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:53:10,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152532 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:53:20,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:53:20,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133256 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:53:20,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:53:20,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154977 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:53:30,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:53:30,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203504 s, 201 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:53:30,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:53:30,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864315 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:53:40,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:53:40,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133689 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:53:40,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:53:40,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162023 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14243::WARNING::2013-03-17 13:53:50,179::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:53:50,184::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 13:53:50,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:53:50,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101712 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:53:50,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:53:50,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848193 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:54:00,186::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:54:00,187::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:54:00,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:54:00,631::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:54:00,646::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:54:00,652::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:54:00,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132615 s, 309 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:54:00,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:54:00,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174404 s, 235 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:54:10,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:54:10,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154782 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:54:10,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:54:10,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145968 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:54:20,760::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:54:20,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000685713 s, 6.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:54:20,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:54:21,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00835444 s, 490 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:54:30,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:54:30,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000719185 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:54:31,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:54:31,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129571 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:54:40,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:54:40,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013421 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:54:41,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:54:41,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157313 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:54:50,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:54:50,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164435 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:54:51,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:54:51,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137466 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14244::WARNING::2013-03-17 13:55:00,714::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:55:00,720::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 13:55:00,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:55:00,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0043105 s, 950 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:55:01,341::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:55:01,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100942 s, 4.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:55:10,724::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:55:10,726::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:55:10,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:55:11,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136551 s, 300 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:55:11,181::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:55:11,187::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:55:11,188::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:55:11,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:55:11,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111176 s, 368 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:55:21,043::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 13:55:21,045::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 13:55:21,045::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:55:21,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000796893 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:55:21,082::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000796893 s, 2.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 13:55:21,083::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 13:55:21,083::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 13:55:21,085::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 13:55:21,086::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:55:21,087::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:55:21,088::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:55:21,088::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:55:21,089::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 13:55:21,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:55:21,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:55:21,514::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:55:21,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:55:21,541::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 13:55:21,562::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 13:55:21,563::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 13:55:21,565::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 13:55:21,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:55:21,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0229276 s, 179 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:55:22,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:55:22,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:55:22,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163128 s, 251 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:55:31,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:55:31,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128433 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:55:32,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:55:32,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904245 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:55:41,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:55:41,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000984946 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:55:42,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:55:42,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762228 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:55:51,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:55:51,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139111 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:55:52,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:55:52,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110431 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:56:01,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:56:01,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159763 s, 256 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:56:02,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:56:02,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120447 s, 3.4 MB/s\n'; <rc> = 0
>Thread-14245::WARNING::2013-03-17 13:56:11,216::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:56:11,219::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 13:56:12,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:56:12,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000971031 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:56:12,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:56:12,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775627 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:56:21,221::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:56:21,222::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:56:21,637::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:56:21,644::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:56:21,645::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:56:22,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:56:22,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146878 s, 279 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:56:22,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:56:22,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159862 s, 256 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:56:32,190::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 13:56:32,192::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 13:56:32,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:56:32,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00149054 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:56:32,238::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00149054 s, 1.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 13:56:32,239::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 13:56:32,239::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 13:56:32,241::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 13:56:32,242::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:56:32,242::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:56:32,243::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:56:32,243::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:56:32,244::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 13:56:32,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:56:32,350::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:56:32,402::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138114 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:56:32,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:56:32,725::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 13:56:32,726::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 13:56:32,728::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 13:56:32,730::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 13:56:32,730::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 13:56:32,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:56:33,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:56:33,099::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:56:33,140::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00736254 s, 556 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:56:42,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:56:42,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827658 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:56:43,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:56:43,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0228283 s, 179 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:56:52,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:56:52,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803512 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:56:53,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:56:53,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135524 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:57:02,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:57:02,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176863 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:57:03,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:57:03,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0218671 s, 187 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:57:12,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:57:12,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.002182 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:57:13,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:57:13,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154599 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14246::WARNING::2013-03-17 13:57:21,670::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:57:21,674::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 13:57:22,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:57:22,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206323 s, 199 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:57:23,524::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:57:23,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193231 s, 212 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:57:31,675::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 13:57:31,677::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:57:31,678::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:57:32,095::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:57:32,102::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:57:32,103::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:57:32,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:57:32,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182501 s, 224 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:57:33,612::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:57:33,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140991 s, 291 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:57:42,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:57:42,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130302 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:57:43,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:57:43,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176944 s, 231 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:57:52,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:57:52,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127568 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:57:53,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:57:53,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139746 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:58:02,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:58:02,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001054 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:58:03,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:58:03,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0070762 s, 579 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:58:12,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:58:12,928::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.020951 s, 196 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:58:13,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:58:13,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797511 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:58:22,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:58:22,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888585 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:58:24,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:58:24,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0407384 s, 101 kB/s\n'; <rc> = 0
>Thread-14247::WARNING::2013-03-17 13:58:32,129::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:58:32,133::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 13:58:32,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:58:33,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819899 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:58:34,156::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:58:34,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917138 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:58:42,135::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:58:42,136::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:58:42,556::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:58:42,563::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:58:42,565::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:58:43,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:58:43,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170113 s, 241 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:58:44,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:58:44,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0316105 s, 130 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:58:53,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:58:53,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129532 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:58:54,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:58:54,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156441 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:59:03,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:59:03,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000974356 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:59:04,416::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:59:04,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000965754 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:59:13,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:59:13,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00249423 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:59:14,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:59:14,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788313 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:59:23,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:59:23,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126385 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:59:24,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:59:24,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180986 s, 226 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 13:59:33,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:59:33,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138775 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:59:34,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:59:34,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083479 s, 4.9 MB/s\n'; <rc> = 0
>Thread-14248::WARNING::2013-03-17 13:59:42,593::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 13:59:42,598::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 13:59:43,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:59:43,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767074 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:59:44,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:59:44,766::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787013 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 13:59:52,602::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 13:59:52,603::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 13:59:53,015::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 13:59:53,021::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 13:59:53,023::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 13:59:53,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 13:59:53,439::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139852 s, 293 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 13:59:54,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 13:59:54,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112004 s, 366 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:00:03,445::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:00:03,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194046 s, 211 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:00:04,899::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:00:04,944::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018522 s, 221 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:00:13,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:00:13,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765196 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:00:14,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:00:15,017::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771601 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:00:23,547::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 14:00:23,548::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 14:00:23,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:00:23,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00241791 s, 847 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:00:23,607::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00241791 s, 847 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 14:00:23,608::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 14:00:23,608::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 14:00:23,609::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 14:00:23,610::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:00:23,611::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:00:23,611::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:00:23,611::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:00:23,611::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:00:23,612::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:00:24,017::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:00:24,022::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:00:24,023::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 14:00:24,025::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 14:00:24,026::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 14:00:24,027::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 14:00:24,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:00:24,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:00:24,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:00:24,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123144 s, 333 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:00:25,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:00:25,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125572 s, 326 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:00:34,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:00:34,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802545 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:00:35,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:00:35,188::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866101 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:00:44,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:00:44,513::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00206601 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:00:45,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:00:45,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159355 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14249::WARNING::2013-03-17 14:00:53,049::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:00:53,053::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:00:54,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:00:54,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825575 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:00:55,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:00:55,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868069 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:01:03,055::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:01:03,056::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:01:03,519::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:01:03,526::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:01:03,528::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:01:04,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:01:04,598::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00638781 s, 641 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:01:05,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:01:05,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01716 s, 239 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:01:14,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:01:14,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783828 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:01:15,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:01:15,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771694 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:01:24,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:01:24,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186466 s, 220 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:01:25,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:01:25,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016921 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:01:34,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:01:34,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873459 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:01:35,618::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 14:01:35,621::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 14:01:35,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:01:35,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00100871 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:01:35,651::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00100871 s, 2.0 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 14:01:35,651::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 14:01:35,652::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 14:01:35,653::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 14:01:35,654::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:01:35,655::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:01:35,656::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:01:35,656::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:01:35,656::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:01:35,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:01:36,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:01:36,097::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:01:36,098::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 14:01:36,101::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 14:01:36,102::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 14:01:36,103::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 14:01:36,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:01:36,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:01:36,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:01:36,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00778247 s, 526 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:01:44,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:01:44,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812104 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:01:46,603::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:01:46,631::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078135 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:01:54,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:01:54,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125919 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:01:56,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:01:56,709::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844416 s, 4.9 MB/s\n'; <rc> = 0
>Thread-14250::WARNING::2013-03-17 14:02:03,559::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:02:03,565::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:02:04,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:02:04,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118887 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:02:06,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:02:06,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828021 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:02:13,576::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:02:13,577::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:02:14,025::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:02:14,031::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:02:14,033::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:02:14,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:02:14,899::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125899 s, 325 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:02:16,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:02:16,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103878 s, 394 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:02:24,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:02:24,950::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00093506 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:02:26,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:02:26,936::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163979 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:02:34,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:02:35,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800002 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:02:36,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:02:37,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864716 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:02:45,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:02:45,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141639 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:02:47,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:02:47,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849267 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:02:55,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:02:55,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00677288 s, 605 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:02:57,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:02:57,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001486 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:03:05,099::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:03:05,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145331 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:03:07,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:03:07,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150333 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14251::WARNING::2013-03-17 14:03:14,062::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:03:14,069::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:03:15,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:03:15,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089147 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:03:17,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:03:17,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848539 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:03:24,085::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 14:03:24,086::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:03:24,087::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:03:24,521::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:03:24,528::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:03:24,529::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:03:25,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:03:25,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.012445 s, 329 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:03:27,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:03:27,402::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0208437 s, 197 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:03:35,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:03:35,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148416 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:03:37,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:03:37,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000950635 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:03:45,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:03:45,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874252 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:03:47,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:03:47,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0220577 s, 186 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:03:55,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:03:55,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00246957 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:03:57,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:03:57,657::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134537 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:04:05,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:04:05,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132654 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:04:07,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:04:07,739::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143216 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:04:15,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:04:15,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135311 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:04:17,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:04:17,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0228258 s, 179 kB/s\n'; <rc> = 0
>Thread-14252::WARNING::2013-03-17 14:04:24,553::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:04:24,557::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:04:25,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:04:25,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780968 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:04:27,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:04:27,936::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0296346 s, 138 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:04:34,559::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:04:34,560::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:04:35,011::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:04:35,018::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:04:35,019::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:04:35,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:04:35,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0432 s, 94.8 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:04:37,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:04:38,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101652 s, 403 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:04:45,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:04:45,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110082 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:04:48,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:04:48,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092631 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:04:55,592::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:04:55,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839144 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:04:58,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:04:58,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156019 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:05:05,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:05:05,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000675262 s, 6.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:05:08,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:05:08,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129192 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:05:15,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:05:15,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916695 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:05:18,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:05:18,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178955 s, 229 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:05:25,700::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 14:05:25,701::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 14:05:25,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:05:25,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000802093 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:05:25,732::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000802093 s, 2.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 14:05:25,733::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 14:05:25,733::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 14:05:25,735::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 14:05:25,743::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:05:25,744::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:05:25,744::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:05:25,745::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:05:25,745::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:05:25,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:05:26,188::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:05:26,193::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:05:26,194::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 14:05:26,196::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 14:05:26,197::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 14:05:26,198::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 14:05:26,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:05:26,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:05:26,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:05:26,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168491 s, 243 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:05:28,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:05:28,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0212322 s, 193 kB/s\n'; <rc> = 0
>Thread-14253::WARNING::2013-03-17 14:05:35,049::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:05:35,054::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:05:36,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:05:36,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832297 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:05:38,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:05:38,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00821091 s, 499 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:05:45,058::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:05:45,059::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:05:45,522::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:05:45,529::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:05:45,530::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:05:46,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:05:46,739::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117197 s, 349 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:05:48,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:05:48,636::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0244653 s, 167 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:05:56,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:05:56,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916373 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:05:58,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:05:58,709::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877811 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:06:06,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:06:06,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150415 s, 272 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:06:08,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:06:08,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151932 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:06:16,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:06:16,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129496 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:06:18,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:06:18,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0374417 s, 109 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:06:26,894::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:06:26,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843987 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:06:28,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:06:28,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845743 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:06:36,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:06:36,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740135 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:06:39,027::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 14:06:39,028::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 14:06:39,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:06:39,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00121206 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:06:39,072::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00121206 s, 1.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 14:06:39,072::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 14:06:39,073::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 14:06:39,074::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 14:06:39,078::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:06:39,078::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:06:39,079::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:06:39,079::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:06:39,080::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:06:39,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:06:39,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:06:39,542::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:06:39,543::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 14:06:39,548::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 14:06:39,549::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 14:06:39,549::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 14:06:39,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:06:39,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:06:39,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:06:39,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113824 s, 360 kB/s\n'; <rc> = 0
>Thread-14254::WARNING::2013-03-17 14:06:45,558::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:06:45,565::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:06:46,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:06:46,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895032 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:06:49,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:06:50,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000964693 s, 4.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:06:55,567::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:06:55,568::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:06:56,069::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:06:56,076::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:06:56,077::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:06:57,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:06:57,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139863 s, 293 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:07:00,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:07:00,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179323 s, 228 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:07:07,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:07:07,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781261 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:07:10,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:07:10,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155604 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:07:17,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:07:17,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145521 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:07:20,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:07:20,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846574 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:07:27,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:07:27,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168412 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:07:30,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:07:30,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892649 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:07:37,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:07:37,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155652 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:07:40,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:07:40,408::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00090081 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:07:47,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:07:47,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142639 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:07:50,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:07:50,483::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894761 s, 4.6 MB/s\n'; <rc> = 0
>Thread-14255::WARNING::2013-03-17 14:07:56,105::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:07:56,112::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:07:57,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:07:57,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0209054 s, 196 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:08:00,532::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:08:00,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840955 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:08:06,115::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:08:06,116::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:08:06,557::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:08:06,567::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:08:06,568::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:08:07,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:08:07,382::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146308 s, 280 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:08:10,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:08:10,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199428 s, 205 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:08:17,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:08:17,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840207 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:08:20,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:08:20,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0267161 s, 153 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:08:27,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:08:27,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851884 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:08:30,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:08:30,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084043 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:08:37,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:08:37,491::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156717 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:08:40,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:08:40,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0267337 s, 153 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:08:47,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:08:47,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144015 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:08:50,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:08:51,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140031 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:08:57,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:08:57,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170066 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:09:01,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:09:01,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146163 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14256::WARNING::2013-03-17 14:09:06,597::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:09:06,601::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:09:07,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:09:07,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870225 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:09:11,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:09:11,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000889241 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:09:16,602::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 14:09:16,603::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:09:16,605::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:09:17,052::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:09:17,059::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:09:17,060::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:09:17,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:09:17,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108734 s, 377 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:09:21,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:09:21,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175407 s, 234 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:09:27,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:09:27,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165099 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:09:31,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:09:31,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206137 s, 199 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:09:37,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:09:37,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166425 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:09:41,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:09:41,450::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142124 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:09:47,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:09:47,809::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157566 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:09:51,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:09:51,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0274733 s, 149 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:09:57,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:09:57,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868354 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:10:01,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:10:01,692::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016531 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:10:07,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:10:07,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830087 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:10:11,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:10:11,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013681 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14257::WARNING::2013-03-17 14:10:17,091::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:10:17,096::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:10:17,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:10:17,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912589 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:10:21,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:10:21,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000750698 s, 5.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:10:27,101::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:10:27,108::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:10:27,565::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:10:27,571::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:10:27,572::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:10:27,932::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 14:10:27,935::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 14:10:27,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:10:27,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0185713 s, 110 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:10:27,985::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0185713 s, 110 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 14:10:27,985::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 14:10:27,985::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 14:10:27,987::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 14:10:27,988::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:10:27,989::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:10:27,989::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:10:27,989::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:10:27,990::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:10:27,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:10:28,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:10:28,360::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:10:28,361::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 14:10:28,364::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 14:10:28,365::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 14:10:28,365::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 14:10:28,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:10:28,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:10:28,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:10:28,766::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125953 s, 325 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:10:31,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:10:32,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0157019 s, 261 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:10:38,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:10:38,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0231683 s, 177 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:10:42,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:10:42,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827991 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:10:48,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:10:48,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00069244 s, 5.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:10:52,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:10:52,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013726 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:10:58,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:10:58,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774308 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:11:02,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:11:02,238::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151236 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:11:08,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:11:08,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139112 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:11:12,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:11:12,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131048 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:11:18,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:11:18,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135658 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:11:22,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:11:22,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125639 s, 3.3 MB/s\n'; <rc> = 0
>Thread-14258::WARNING::2013-03-17 14:11:27,607::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:11:27,611::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:11:28,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:11:29,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000689885 s, 5.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:11:32,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:11:32,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839209 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:11:37,613::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:11:37,614::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:11:38,096::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:11:38,103::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:11:38,104::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:11:39,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:11:39,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193206 s, 212 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:11:42,504::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 14:11:42,509::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 14:11:42,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:11:42,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0149951 s, 137 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:11:42,553::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0149951 s, 137 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 14:11:42,553::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 14:11:42,554::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 14:11:42,555::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 14:11:42,556::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:11:42,557::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:11:42,557::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:11:42,558::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:11:42,558::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:11:42,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:11:43,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:11:43,005::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:11:43,006::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 14:11:43,010::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 14:11:43,011::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 14:11:43,011::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 14:11:43,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:11:43,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:11:43,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:11:43,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136032 s, 301 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:11:49,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:11:49,132::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774102 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:11:53,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:11:53,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770824 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:11:59,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:11:59,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00196524 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:12:03,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:12:03,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833248 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:12:09,178::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:12:09,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0219014 s, 187 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:12:13,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:12:13,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834201 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:12:19,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:12:19,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783333 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:12:23,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:12:23,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103644 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:12:29,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:12:29,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836031 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:12:33,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:12:33,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163988 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14259::WARNING::2013-03-17 14:12:38,132::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:12:38,136::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:12:39,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:12:39,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833471 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:12:43,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:12:43,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00096014 s, 4.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:12:48,138::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:12:48,139::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:12:48,593::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:12:48,601::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:12:48,602::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:12:49,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:12:49,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010089 s, 406 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:12:53,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:12:53,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177253 s, 231 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:12:59,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:12:59,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142445 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:13:04,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:13:04,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119194 s, 344 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:13:09,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:13:09,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077334 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:13:14,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:13:14,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157867 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:13:19,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:13:19,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137836 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:13:24,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:13:24,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112318 s, 365 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:13:29,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:13:29,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0009837 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:13:34,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:13:34,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872646 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:13:39,606::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:13:39,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773438 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:13:44,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:13:44,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000723693 s, 5.7 MB/s\n'; <rc> = 0
>Thread-14260::WARNING::2013-03-17 14:13:48,630::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:13:48,636::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:13:49,647::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:13:49,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000985959 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:13:54,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:13:54,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076635 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:13:58,639::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:13:58,641::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:13:59,088::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:13:59,096::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:13:59,097::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:13:59,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:13:59,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0300149 s, 136 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:14:04,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:14:04,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171593 s, 239 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:14:09,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:14:09,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014146 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:14:14,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:14:14,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853405 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:14:19,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:14:19,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108902 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:14:24,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:14:24,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855257 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:14:29,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:14:29,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179874 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:14:34,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:14:34,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01292 s, 317 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:14:39,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:14:39,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00618519 s, 662 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:14:44,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:14:44,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876274 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:14:49,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:14:49,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150345 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:14:54,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:14:54,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814134 s, 5.0 MB/s\n'; <rc> = 0
>Thread-14261::WARNING::2013-03-17 14:14:59,130::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:14:59,135::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:14:59,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:15:00,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182312 s, 225 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:15:04,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:15:05,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901776 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:15:09,137::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 14:15:09,140::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:15:09,141::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:15:09,593::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:15:09,601::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:15:09,602::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:15:10,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:15:10,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165088 s, 248 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:15:15,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:15:15,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0307499 s, 133 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:15:20,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:15:20,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184294 s, 222 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:15:25,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:15:25,188::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747706 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:15:30,143::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 14:15:30,145::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 14:15:30,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:15:30,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00123209 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:15:30,187::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00123209 s, 1.7 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 14:15:30,187::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 14:15:30,188::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 14:15:30,189::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 14:15:30,190::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:15:30,191::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:15:30,191::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:15:30,192::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:15:30,192::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:15:30,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:15:30,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:15:30,614::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:15:30,615::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 14:15:30,617::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 14:15:30,618::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 14:15:30,619::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 14:15:30,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:15:30,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:15:30,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:15:31,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177825 s, 230 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:15:35,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:15:35,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112194 s, 365 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:15:41,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:15:41,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076532 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:15:45,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:15:45,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897922 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:15:51,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:15:51,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000934676 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:15:55,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:15:55,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150424 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:16:01,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:16:01,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148803 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:16:05,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:16:05,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132094 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14262::WARNING::2013-03-17 14:16:09,637::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:16:09,643::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:16:11,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:16:11,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902211 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:16:15,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:16:15,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757116 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:16:19,646::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:16:19,647::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:16:20,087::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:16:20,093::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:16:20,095::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:16:21,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:16:21,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184586 s, 222 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:16:25,612::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:16:25,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119173 s, 344 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:16:31,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:16:31,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00350368 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:16:35,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:16:35,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000939981 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:16:41,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:16:41,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144147 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:16:45,769::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 14:16:45,771::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 14:16:45,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:16:45,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000863674 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:16:45,800::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000863674 s, 2.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 14:16:45,800::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 14:16:45,801::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 14:16:45,802::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 14:16:45,803::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:16:45,804::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:16:45,805::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:16:45,805::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:16:45,805::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:16:45,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:16:46,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:16:46,237::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:16:46,238::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 14:16:46,241::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 14:16:46,243::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 14:16:46,243::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 14:16:46,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:16:46,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:16:46,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:16:46,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00968625 s, 423 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:16:51,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:16:51,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859358 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:16:56,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:16:56,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171189 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:17:01,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:17:01,408::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891757 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:17:06,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:17:06,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132522 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:17:11,414::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:17:11,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777472 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:17:16,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:17:16,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920849 s, 4.4 MB/s\n'; <rc> = 0
>Thread-14263::WARNING::2013-03-17 14:17:20,120::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:17:20,124::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:17:21,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:17:21,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829125 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:17:26,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:17:26,973::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781334 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:17:30,127::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:17:30,128::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:17:30,567::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:17:30,574::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:17:30,575::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:17:31,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:17:31,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140308 s, 292 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:17:37,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:17:37,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128055 s, 320 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:17:41,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:17:41,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118448 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:17:47,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:17:47,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153161 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:17:51,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:17:51,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873163 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:17:57,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:17:57,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116274 s, 352 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:18:01,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:18:01,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764405 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:18:07,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:18:07,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136377 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:18:11,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:18:11,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199606 s, 205 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:18:17,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:18:17,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146316 s, 280 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:18:21,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:18:21,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0217445 s, 188 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:18:27,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:18:27,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791868 s, 5.2 MB/s\n'; <rc> = 0
>Thread-14264::WARNING::2013-03-17 14:18:30,600::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:18:30,604::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:18:31,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:18:31,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860326 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:18:37,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:18:37,598::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0241661 s, 169 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:18:40,606::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:18:40,607::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:18:41,104::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:18:41,111::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:18:41,113::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:18:41,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:18:41,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125603 s, 326 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:18:47,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:18:47,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109082 s, 375 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:18:51,875::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:18:51,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100362 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:18:57,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:18:57,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00240903 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:19:01,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:19:01,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881018 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:19:07,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:19:07,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00193805 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:19:11,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:19:11,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153565 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:19:17,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:19:17,912::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163964 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:19:21,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:19:22,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843523 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:19:27,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:19:27,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076333 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:19:32,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:19:32,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075383 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:19:38,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:19:38,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155987 s, 263 kB/s\n'; <rc> = 0
>Thread-14265::WARNING::2013-03-17 14:19:41,159::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:19:41,164::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:19:42,108::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:19:42,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000709154 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:19:48,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:19:48,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766761 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:19:51,168::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:19:51,170::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:19:51,608::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:19:51,615::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:19:51,616::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:19:52,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:19:52,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019051 s, 215 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:19:58,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:19:58,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187133 s, 219 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:20:02,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:20:02,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139067 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:20:08,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:20:08,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153188 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:20:12,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:20:12,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785318 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:20:18,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:20:18,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.023596 s, 174 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:20:22,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:20:22,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133527 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:20:28,471::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:20:28,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873808 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:20:32,327::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 14:20:32,328::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 14:20:32,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:20:32,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000908734 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:20:32,359::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000908734 s, 2.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 14:20:32,359::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 14:20:32,359::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 14:20:32,361::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 14:20:32,362::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:20:32,363::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:20:32,363::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:20:32,363::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:20:32,364::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:20:32,365::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:20:32,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:20:32,779::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:20:32,780::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 14:20:32,783::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 14:20:32,784::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 14:20:32,785::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 14:20:32,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:20:33,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:20:33,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:20:33,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152685 s, 268 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:20:38,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:20:38,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00865641 s, 473 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:20:43,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:20:43,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855522 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:20:48,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:20:48,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795005 s, 5.2 MB/s\n'; <rc> = 0
>Thread-14266::WARNING::2013-03-17 14:20:51,644::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:20:51,647::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:20:53,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:20:53,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0216048 s, 190 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:20:58,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:20:58,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144686 s, 2.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:21:01,648::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 14:21:01,650::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:21:01,651::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:21:02,140::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:21:02,146::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:21:02,148::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:21:03,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:21:03,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160753 s, 255 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:21:08,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:21:08,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174401 s, 235 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:21:13,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:21:13,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764322 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:21:18,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:21:18,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160271 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:21:23,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:21:23,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193065 s, 212 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:21:28,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:21:29,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135338 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:21:33,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:21:33,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0281413 s, 146 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:21:39,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:21:39,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161494 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:21:43,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:21:43,552::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014076 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:21:49,145::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 14:21:49,146::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 14:21:49,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:21:49,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000825198 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:21:49,178::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000825198 s, 2.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 14:21:49,178::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 14:21:49,179::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 14:21:49,181::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 14:21:49,182::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:21:49,183::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:21:49,184::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:21:49,184::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:21:49,184::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:21:49,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:21:49,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:21:49,628::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:21:49,629::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 14:21:49,632::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 14:21:49,633::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 14:21:49,633::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 14:21:49,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:21:50,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:21:50,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:21:50,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0241545 s, 170 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:21:53,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:21:53,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134864 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:22:00,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:22:00,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740462 s, 5.5 MB/s\n'; <rc> = 0
>Thread-14267::WARNING::2013-03-17 14:22:02,188::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:22:02,194::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:22:03,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:22:03,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869809 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:22:10,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:22:10,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862084 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:22:12,197::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:22:12,198::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:22:12,656::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:22:12,663::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:22:12,664::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:22:13,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:22:13,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135336 s, 303 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:22:20,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:22:20,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00801485 s, 511 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:22:23,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:22:23,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0205068 s, 200 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:22:30,345::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:22:30,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797579 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:22:33,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:22:33,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159133 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:22:40,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:22:40,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0219714 s, 186 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:22:43,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:22:43,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142228 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:22:50,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:22:50,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170226 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:22:53,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:22:53,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900706 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:23:00,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:23:00,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152318 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:23:03,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:23:03,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134983 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:23:10,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:23:10,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0231955 s, 177 kB/s\n'; <rc> = 0
>Thread-14268::WARNING::2013-03-17 14:23:12,693::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:23:12,697::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:23:13,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:23:13,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133097 s, 308 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:23:20,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:23:20,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0218147 s, 188 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:23:22,699::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:23:22,701::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:23:23,135::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:23:23,142::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:23:23,144::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:23:24,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:23:24,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.038931 s, 105 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:23:30,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:23:30,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173794 s, 236 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:23:34,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:23:34,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785193 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:23:41,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:23:41,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874143 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:23:44,117::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:23:44,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181774 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:23:51,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:23:51,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.023505 s, 174 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:23:54,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:23:54,195::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079244 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:24:01,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:24:01,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000717576 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:24:04,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:24:04,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108107 s, 379 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:24:11,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:24:11,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139567 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:24:14,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:24:14,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783208 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:24:21,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:24:21,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132082 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14269::WARNING::2013-03-17 14:24:23,193::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:24:23,197::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:24:24,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:24:24,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740315 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:24:31,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:24:31,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839189 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:24:33,201::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:24:33,202::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:24:33,792::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:24:33,799::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:24:33,801::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:24:34,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:24:34,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133847 s, 306 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:24:41,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:24:41,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162961 s, 251 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:24:44,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:24:44,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758554 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:24:51,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:24:51,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785854 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:24:54,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:24:54,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932257 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:25:01,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:25:01,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0236832 s, 173 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:25:04,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:25:04,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103741 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:25:11,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:25:11,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0252399 s, 162 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:25:14,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:25:14,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153513 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:25:21,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:25:21,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768323 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:25:24,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:25:24,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828006 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:25:31,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:25:32,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119983 s, 3.4 MB/s\n'; <rc> = 0
>Thread-14270::WARNING::2013-03-17 14:25:33,827::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:25:33,831::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:25:34,584::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 14:25:34,586::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 14:25:34,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:25:34,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00077613 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:25:34,615::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00077613 s, 2.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 14:25:34,616::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 14:25:34,616::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 14:25:34,618::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 14:25:34,619::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:25:34,620::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:25:34,621::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:25:34,621::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:25:34,622::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:25:34,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:25:35,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:25:35,072::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:25:35,073::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 14:25:35,075::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 14:25:35,077::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 14:25:35,077::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 14:25:35,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:25:35,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:25:35,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:25:35,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142766 s, 287 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:25:42,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:25:42,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183378 s, 223 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:25:43,833::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:25:43,834::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:25:44,384::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:25:44,391::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:25:44,392::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:25:45,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:25:45,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0454649 s, 90.1 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:25:52,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:25:52,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0215136 s, 190 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:25:55,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:25:55,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019784 s, 207 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:26:02,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:26:02,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00191237 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:26:05,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:26:05,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186907 s, 219 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:26:12,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:26:12,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144006 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:26:15,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:26:15,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149079 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:26:22,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:26:22,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0246934 s, 166 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:26:25,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:26:25,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127936 s, 320 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:26:32,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:26:32,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174176 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:26:35,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:26:35,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854214 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:26:42,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:26:42,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119625 s, 3.4 MB/s\n'; <rc> = 0
>Thread-14271::WARNING::2013-03-17 14:26:44,417::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:26:44,423::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:26:45,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:26:45,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000998389 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:26:52,699::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 14:26:52,700::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 14:26:52,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:26:52,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000904246 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:26:52,728::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000904246 s, 2.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 14:26:52,729::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 14:26:52,729::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 14:26:52,731::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 14:26:52,732::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:26:52,732::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:26:52,733::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:26:52,733::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:26:52,734::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:26:52,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:26:53,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:26:53,140::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:26:53,140::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 14:26:53,143::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 14:26:53,144::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 14:26:53,144::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 14:26:53,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:26:53,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:26:53,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:26:53,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00285703 s, 1.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:26:54,427::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 14:26:54,428::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:26:54,429::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:26:54,933::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:26:54,940::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:26:54,941::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:26:55,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:26:55,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104553 s, 392 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:27:03,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:27:03,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188323 s, 217 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:27:05,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:27:05,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092599 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:27:13,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:27:13,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016408 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:27:15,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:27:15,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823218 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:27:23,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:27:23,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0220215 s, 186 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:27:26,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:27:26,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845594 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:27:33,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:27:33,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135939 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:27:36,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:27:36,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749698 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:27:43,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:27:43,963::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906806 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:27:46,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:27:46,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145351 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:27:54,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:27:54,140::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0260168 s, 157 kB/s\n'; <rc> = 0
>Thread-14272::WARNING::2013-03-17 14:27:54,968::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:27:54,972::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:27:56,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:27:56,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117278 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:28:04,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:28:04,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000918848 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:28:04,978::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:28:04,979::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:28:05,526::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:28:05,534::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:28:05,535::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:28:06,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:28:06,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.020201 s, 203 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:28:14,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:28:14,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0200904 s, 204 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:28:16,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:28:16,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150704 s, 272 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:28:24,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:28:24,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077757 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:28:26,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:28:26,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853389 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:28:34,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:28:34,667::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820569 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:28:36,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:28:36,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000940716 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:28:44,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:28:44,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000731309 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:28:46,365::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:28:46,409::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174636 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:28:54,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:28:54,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822542 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:28:56,414::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:28:56,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120919 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:29:04,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:29:05,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846952 s, 4.8 MB/s\n'; <rc> = 0
>Thread-14273::WARNING::2013-03-17 14:29:05,576::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:29:05,580::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:29:06,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:29:06,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000999166 s, 4.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:29:15,583::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:29:15,584::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:29:15,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:29:15,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00995951 s, 411 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:29:16,116::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:29:16,122::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:29:16,123::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:29:16,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:29:16,548::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166374 s, 246 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:29:25,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:29:25,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865907 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:29:26,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:29:26,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808841 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:29:35,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:29:35,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0096699 s, 424 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:29:36,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:29:36,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000974041 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:29:45,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:29:45,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817912 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:29:46,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:29:46,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780221 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:29:55,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:29:56,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142819 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:29:56,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:29:56,709::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000940682 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:30:06,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:30:06,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0216853 s, 189 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:30:06,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:30:06,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00285135 s, 1.4 MB/s\n'; <rc> = 0
>Thread-14274::WARNING::2013-03-17 14:30:16,158::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:30:16,163::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 14:30:16,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:30:16,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870883 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:30:16,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:30:16,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806225 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:30:26,166::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:30:26,168::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:30:26,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:30:26,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0290076 s, 141 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:30:26,687::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:30:26,693::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:30:26,694::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:30:26,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:30:26,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182423 s, 225 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:30:36,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:30:36,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761318 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:30:36,928::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 14:30:36,930::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 14:30:36,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:30:36,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000730486 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:30:36,969::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000730486 s, 2.8 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 14:30:36,969::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 14:30:36,969::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 14:30:36,971::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 14:30:36,972::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:30:36,973::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:30:36,973::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:30:36,974::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:30:36,974::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:30:36,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:30:37,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:30:37,410::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:30:37,411::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 14:30:37,414::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 14:30:37,415::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 14:30:37,415::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 14:30:37,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:30:37,801::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:30:37,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:30:37,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206508 s, 198 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:30:46,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:30:46,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019635 s, 209 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:30:47,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:30:47,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141627 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:30:56,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:30:56,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135794 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:30:57,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:30:57,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828804 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:31:06,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:31:06,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142685 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:31:07,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:31:07,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765904 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:31:16,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:31:16,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016627 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:31:17,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:31:18,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162076 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14275::WARNING::2013-03-17 14:31:26,723::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:31:26,725::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 14:31:26,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:31:26,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163858 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:31:28,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:31:28,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861689 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:31:36,728::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:31:36,730::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:31:36,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:31:36,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125766 s, 326 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:31:37,257::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:31:37,263::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:31:37,264::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:31:38,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:31:38,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121174 s, 338 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:31:47,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:31:47,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827769 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:31:48,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:31:48,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173568 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:31:57,094::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 14:31:57,096::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 14:31:57,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:31:57,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0233727 s, 87.6 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:31:57,151::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0233727 s, 87.6 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 14:31:57,151::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 14:31:57,152::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 14:31:57,154::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 14:31:57,155::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:31:57,156::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:31:57,157::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:31:57,157::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:31:57,158::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:31:57,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:31:57,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:31:57,582::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:31:57,583::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 14:31:57,586::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 14:31:57,587::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 14:31:57,587::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 14:31:57,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:31:57,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:31:57,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:31:58,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151437 s, 270 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:31:58,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:31:58,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145727 s, 281 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:32:08,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:32:08,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785358 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:32:08,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:32:08,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823037 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:32:18,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:32:18,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841518 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:32:18,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:32:18,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151022 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:32:28,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:32:28,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000707433 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:32:28,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:32:28,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904585 s, 4.5 MB/s\n'; <rc> = 0
>Thread-14276::WARNING::2013-03-17 14:32:37,290::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:32:37,294::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:32:38,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:32:38,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:32:38,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129955 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:32:38,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130191 s, 3.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:32:47,295::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 14:32:47,297::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:32:47,298::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:32:47,894::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:32:47,900::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:32:47,902::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:32:48,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:32:48,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:32:48,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766811 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:32:48,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173196 s, 236 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:32:58,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:32:58,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:32:58,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00228464 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:32:58,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204235 s, 201 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:33:08,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:33:08,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:33:08,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012098 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:33:08,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115626 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:33:18,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:33:18,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:33:19,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111194 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:33:19,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0205497 s, 199 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:33:29,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:33:29,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:33:29,188::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888421 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:33:29,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126942 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:33:39,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:33:39,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:33:39,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112007 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:33:39,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00706329 s, 580 kB/s\n'; <rc> = 0
>Thread-14277::WARNING::2013-03-17 14:33:47,931::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:33:47,936::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:33:49,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:33:49,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:33:49,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822044 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:33:49,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138495 s, 3.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:33:57,939::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:33:57,940::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:33:58,403::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:33:58,409::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:33:58,411::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:33:59,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:33:59,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:33:59,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194036 s, 211 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:33:59,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121591 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:34:09,556::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:34:09,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085224 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:34:09,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:34:09,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138145 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:34:19,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:34:19,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000956211 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:34:19,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:34:19,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846639 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:34:29,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:34:29,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852392 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:34:29,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:34:29,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836537 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:34:39,742::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:34:39,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000675427 s, 6.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:34:39,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:34:39,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140961 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:34:49,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:34:49,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00204594 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:34:49,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:34:49,958::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0018098 s, 2.3 MB/s\n'; <rc> = 0
>Thread-14278::WARNING::2013-03-17 14:34:58,438::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:34:58,444::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:34:59,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:34:59,944::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843654 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:35:00,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:35:00,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852279 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:35:08,446::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:35:08,447::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:35:08,924::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:35:08,930::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:35:08,931::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:35:09,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:35:10,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0257105 s, 159 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:35:10,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:35:10,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785422 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:35:20,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:35:20,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0070998 s, 577 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:35:20,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:35:20,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836241 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:35:30,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:35:30,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081799 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:35:30,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:35:30,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000982797 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:35:40,167::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 14:35:40,168::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 14:35:40,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:35:40,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000731388 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:35:40,195::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000731388 s, 2.8 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 14:35:40,195::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 14:35:40,195::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 14:35:40,197::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 14:35:40,198::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:35:40,199::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:35:40,199::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:35:40,200::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:35:40,200::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:35:40,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:35:40,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:35:40,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159685 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:35:40,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:35:40,675::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:35:40,676::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 14:35:40,678::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 14:35:40,679::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 14:35:40,680::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 14:35:40,681::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:35:41,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:35:41,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:35:41,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00906696 s, 452 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:35:50,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:35:50,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145824 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:35:51,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:35:51,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893611 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:36:00,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:36:00,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178584 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:36:01,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:36:01,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00278239 s, 1.5 MB/s\n'; <rc> = 0
>Thread-14279::WARNING::2013-03-17 14:36:08,971::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:36:08,977::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 14:36:10,560::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:36:10,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000679935 s, 6.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:36:11,171::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:36:11,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000994312 s, 4.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:36:18,979::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:36:18,981::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:36:19,415::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:36:19,422::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:36:19,423::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:36:20,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:36:20,649::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891209 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:36:21,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:36:21,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0195187 s, 210 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:36:30,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:36:30,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868133 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:36:31,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:36:31,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0196275 s, 209 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:36:40,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:36:40,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084015 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:36:41,324::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:36:41,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0210146 s, 195 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:36:50,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:36:50,899::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148255 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:36:51,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:36:51,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174684 s, 234 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:37:00,937::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 14:37:00,938::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 14:37:00,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:37:00,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000867281 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:37:00,972::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000867281 s, 2.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 14:37:00,973::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 14:37:00,973::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 14:37:00,975::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 14:37:00,976::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:37:00,977::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:37:00,978::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:37:00,978::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:37:00,980::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:37:00,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:37:01,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:37:01,416::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:37:01,418::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 14:37:01,422::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 14:37:01,423::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 14:37:01,424::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 14:37:01,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:37:01,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:37:01,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00657208 s, 623 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:37:01,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:37:01,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:37:01,963::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014025 s, 292 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:37:11,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:37:11,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00508287 s, 806 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:37:12,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:37:12,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160898 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14280::WARNING::2013-03-17 14:37:19,452::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:37:19,455::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:37:21,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:37:21,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197847 s, 207 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:37:22,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:37:22,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0329921 s, 124 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:37:29,457::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:37:29,458::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:37:29,947::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:37:29,954::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:37:29,955::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:37:31,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:37:31,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0201831 s, 203 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:37:32,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:37:32,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0261911 s, 156 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:37:41,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:37:41,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.020441 s, 200 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:37:42,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:37:42,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140712 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:37:51,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:37:51,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170109 s, 241 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:37:52,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:37:52,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768623 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:38:01,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:38:01,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190235 s, 215 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:38:02,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:38:02,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152195 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:38:11,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:38:11,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173387 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:38:12,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:38:12,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149474 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:38:21,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:38:21,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00892255 s, 459 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:38:22,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:38:22,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147734 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14281::WARNING::2013-03-17 14:38:29,979::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:38:29,983::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:38:31,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:38:31,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917906 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:38:32,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:38:32,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181769 s, 2.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:38:39,984::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 14:38:39,986::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:38:39,987::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:38:40,428::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:38:40,435::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:38:40,436::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:38:42,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:38:42,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171452 s, 239 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:38:42,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:38:42,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793267 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:38:52,060::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:38:52,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880134 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:38:52,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:38:52,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084901 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:39:02,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:39:02,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102028 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:39:02,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:39:02,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00243784 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:39:12,128::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:39:12,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000721025 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:39:12,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:39:13,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146948 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:39:22,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:39:22,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190248 s, 215 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:39:23,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:39:23,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157489 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:39:32,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:39:32,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878607 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:39:33,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:39:33,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141979 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14282::WARNING::2013-03-17 14:39:40,464::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:39:40,469::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:39:42,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:39:42,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000946969 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:39:43,230::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:39:43,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808693 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:39:50,473::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:39:50,474::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:39:50,916::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:39:50,923::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:39:50,924::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:39:52,305::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:39:52,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131513 s, 311 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:39:53,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:39:53,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015432 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:40:02,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:40:02,401::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169072 s, 242 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:40:03,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:40:03,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091234 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:40:12,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:40:12,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832418 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:40:13,445::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:40:13,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162262 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:40:22,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:40:22,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852841 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:40:23,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:40:23,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153305 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:40:32,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:40:32,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858613 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:40:33,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:40:33,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155652 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:40:42,514::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 14:40:42,516::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 14:40:42,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:40:42,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.001399 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:40:42,563::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.001399 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 14:40:42,563::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 14:40:42,563::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 14:40:42,565::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 14:40:42,566::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:40:42,567::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:40:42,567::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:40:42,567::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:40:42,568::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:40:42,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:40:42,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:40:43,003::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:40:43,004::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 14:40:43,006::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 14:40:43,008::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 14:40:43,008::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 14:40:43,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:40:43,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:40:43,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:40:43,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174463 s, 235 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:40:43,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:40:43,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014984 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14283::WARNING::2013-03-17 14:40:50,964::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:40:50,969::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:40:53,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:40:53,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155362 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:40:53,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:40:53,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160849 s, 2.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:41:00,973::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:41:00,974::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:41:01,451::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:41:01,458::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:41:01,460::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:41:03,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:41:03,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155703 s, 263 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:41:03,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:41:03,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790879 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:41:13,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:41:13,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173055 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:41:13,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:41:13,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152525 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:41:23,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:41:23,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014429 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:41:24,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:41:24,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147569 s, 278 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:41:33,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:41:33,671::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162067 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:41:34,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:41:34,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832075 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:41:43,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:41:43,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000694979 s, 5.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:41:44,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:41:44,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153599 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:41:53,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:41:53,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155144 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:41:54,260::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:41:54,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764018 s, 5.4 MB/s\n'; <rc> = 0
>Thread-14284::WARNING::2013-03-17 14:42:01,487::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:42:01,493::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:42:03,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:42:03,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183717 s, 223 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:42:04,343::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 14:42:04,345::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 14:42:04,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:42:04,372::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00096893 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:42:04,373::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00096893 s, 2.1 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 14:42:04,374::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 14:42:04,374::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 14:42:04,376::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 14:42:04,377::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:42:04,377::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:42:04,378::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:42:04,378::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:42:04,379::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:42:04,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:42:04,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:42:04,808::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:42:04,809::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 14:42:04,811::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 14:42:04,813::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 14:42:04,813::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 14:42:04,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:42:05,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:42:05,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:42:05,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0217852 s, 188 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:42:11,496::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:42:11,497::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:42:11,981::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:42:11,987::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:42:11,988::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:42:13,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:42:13,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126504 s, 324 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:42:15,271::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:42:15,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746611 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:42:23,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:42:23,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842159 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:42:25,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:42:25,366::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763737 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:42:33,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:42:33,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156448 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:42:35,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:42:35,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010024 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:42:43,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:42:44,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134427 s, 305 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:42:45,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:42:45,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00480571 s, 852 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:42:54,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:42:54,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141062 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:42:55,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:42:55,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829033 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:43:04,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:43:04,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166013 s, 247 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:43:05,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:43:05,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0239122 s, 171 kB/s\n'; <rc> = 0
>Thread-14285::WARNING::2013-03-17 14:43:12,015::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:43:12,027::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:43:14,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:43:14,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759596 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:43:15,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:43:15,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.02234 s, 183 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:43:22,031::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:43:22,032::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:43:22,482::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:43:22,488::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:43:22,490::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:43:24,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:43:24,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147858 s, 277 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:43:25,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:43:25,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162854 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:43:34,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:43:34,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820353 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:43:35,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:43:35,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866296 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:43:44,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:43:44,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120562 s, 340 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:43:45,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:43:46,017::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852734 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:43:54,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:43:54,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115016 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:43:56,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:43:56,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835517 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:44:04,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:44:04,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892158 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:44:06,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:44:06,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135874 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:44:14,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:44:14,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744782 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:44:16,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:44:16,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782809 s, 5.2 MB/s\n'; <rc> = 0
>Thread-14286::WARNING::2013-03-17 14:44:22,517::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:44:22,521::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:44:24,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:44:24,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00184972 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:44:26,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:44:26,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000711336 s, 5.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:44:32,522::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 14:44:32,524::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:44:32,525::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:44:32,981::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:44:32,987::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:44:32,988::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:44:34,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:44:34,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130402 s, 314 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:44:36,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:44:36,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151863 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:44:44,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:44:44,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802314 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:44:46,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:44:46,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909794 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:44:54,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:44:54,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164733 s, 249 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:44:56,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:44:56,562::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00184699 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:45:04,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:45:04,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013318 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:45:06,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:45:06,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142769 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:45:14,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:45:14,744::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00226242 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:45:16,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:45:16,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148001 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:45:24,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:45:24,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157414 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:45:26,760::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:45:26,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102295 s, 4.0 MB/s\n'; <rc> = 0
>Thread-14287::WARNING::2013-03-17 14:45:33,017::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:45:33,022::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:45:34,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:45:34,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858613 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:45:36,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:45:36,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000938034 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:45:43,025::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:45:43,026::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:45:43,468::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:45:43,474::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:45:43,476::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:45:44,836::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 14:45:44,845::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 14:45:44,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:45:44,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0125259 s, 164 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:45:44,893::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0125259 s, 164 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 14:45:44,894::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 14:45:44,894::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 14:45:44,896::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 14:45:44,898::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:45:44,898::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:45:44,899::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:45:44,899::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:45:44,900::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:45:44,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:45:45,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:45:45,349::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:45:45,350::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 14:45:45,353::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 14:45:45,354::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 14:45:45,354::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 14:45:45,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:45:45,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:45:45,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:45:45,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126967 s, 323 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:45:46,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:45:46,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759777 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:45:55,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:45:55,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817345 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:45:56,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:45:57,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000713455 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:46:05,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:46:05,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168266 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:46:07,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:46:07,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0244773 s, 167 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:46:15,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:46:15,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00097914 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:46:17,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:46:17,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834007 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:46:25,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:46:25,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834562 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:46:27,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:46:27,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000718315 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:46:35,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:46:35,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793757 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:46:37,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:46:37,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142833 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14288::WARNING::2013-03-17 14:46:43,501::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:46:43,506::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:46:45,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:46:46,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823973 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:46:47,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:46:47,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00094447 s, 4.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:46:53,508::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:46:53,509::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:46:53,944::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:46:53,950::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:46:53,952::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:46:56,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:46:56,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019523 s, 210 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:46:57,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:46:57,450::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772643 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:47:06,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:47:06,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000961647 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:47:07,490::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 14:47:07,492::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 14:47:07,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:47:07,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000763973 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:47:07,519::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000763973 s, 2.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 14:47:07,520::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 14:47:07,520::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 14:47:07,522::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 14:47:07,523::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:47:07,523::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:47:07,524::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:47:07,524::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:47:07,525::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:47:07,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:47:07,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:47:07,943::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:47:07,944::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 14:47:07,946::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 14:47:07,947::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 14:47:07,948::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 14:47:07,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:47:08,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:47:08,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:47:08,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122323 s, 335 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:47:16,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:47:16,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798254 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:47:18,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:47:18,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788617 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:47:26,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:47:26,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138471 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:47:28,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:47:28,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146222 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:47:36,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:47:36,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807764 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:47:38,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:47:38,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109399 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:47:46,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:47:46,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162994 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:47:48,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:47:48,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879177 s, 4.7 MB/s\n'; <rc> = 0
>Thread-14289::WARNING::2013-03-17 14:47:53,981::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:47:53,984::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:47:56,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:47:56,333::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155807 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:47:58,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:47:58,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795734 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:48:03,986::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:48:03,988::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:48:04,431::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:48:04,438::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:48:04,439::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:48:06,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:48:06,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155488 s, 263 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:48:08,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:48:08,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749998 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:48:16,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:48:16,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085919 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:48:18,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:48:18,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160253 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:48:26,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:48:26,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783526 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:48:28,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:48:28,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000986559 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:48:36,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:48:36,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000946449 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:48:38,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:48:39,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00180725 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:48:46,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:48:46,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000739111 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:48:49,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:48:49,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119207 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:48:56,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:48:56,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148337 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:48:59,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:48:59,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103147 s, 4.0 MB/s\n'; <rc> = 0
>Thread-14290::WARNING::2013-03-17 14:49:04,469::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:49:04,472::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:49:06,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:49:06,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740984 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:49:09,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:49:09,260::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851701 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:49:14,474::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:49:14,476::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:49:14,892::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:49:14,899::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:49:14,900::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:49:16,625::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:49:16,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190033 s, 216 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:49:19,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:49:19,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768968 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:49:26,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:49:26,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00098858 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:49:29,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:49:29,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840595 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:49:36,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:49:36,739::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808225 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:49:39,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:49:39,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.022152 s, 185 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:49:46,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:49:46,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126833 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:49:49,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:49:49,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789203 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:49:56,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:49:56,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175496 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:49:59,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:49:59,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00184824 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:50:06,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:50:06,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0031771 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:50:09,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:50:09,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00186242 s, 2.2 MB/s\n'; <rc> = 0
>Thread-14291::WARNING::2013-03-17 14:50:14,947::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:50:14,951::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:50:16,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:50:16,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809519 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:50:19,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:50:19,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836168 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:50:24,953::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 14:50:24,954::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:50:24,955::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:50:25,411::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:50:25,418::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:50:25,419::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:50:26,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:50:26,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161844 s, 253 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:50:29,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:50:29,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00449989 s, 910 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:50:36,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:50:37,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875574 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:50:39,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:50:39,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137749 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:50:47,014::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 14:50:47,015::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 14:50:47,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:50:47,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00130145 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:50:47,057::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00130145 s, 1.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 14:50:47,057::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 14:50:47,058::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 14:50:47,059::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 14:50:47,060::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:50:47,062::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:50:47,062::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:50:47,063::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:50:47,063::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:50:47,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:50:47,491::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:50:47,496::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:50:47,498::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 14:50:47,501::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 14:50:47,503::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 14:50:47,503::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 14:50:47,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:50:47,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:50:47,859::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:50:47,907::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019237 s, 213 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:50:49,993::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:50:50,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857847 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:50:57,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:50:57,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842023 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:51:00,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:51:00,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148686 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:51:07,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:51:07,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120739 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:51:10,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:51:10,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752041 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:51:17,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:51:18,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818117 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:51:20,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:51:20,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00206111 s, 2.0 MB/s\n'; <rc> = 0
>Thread-14292::WARNING::2013-03-17 14:51:25,445::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:51:25,449::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:51:28,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:51:28,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904047 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:51:30,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:51:30,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175573 s, 2.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:51:35,451::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:51:35,453::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:51:35,986::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:51:35,993::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:51:35,994::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:51:38,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:51:38,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01544 s, 265 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:51:40,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:51:40,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857266 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:51:48,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:51:48,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777324 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:51:50,492::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:51:50,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082874 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:51:58,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:51:58,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872617 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:52:00,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:52:00,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143673 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:52:08,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:52:08,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137018 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:52:10,673::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 14:52:10,675::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 14:52:10,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:52:10,738::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0230495 s, 88.9 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:52:10,740::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0230495 s, 88.9 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 14:52:10,741::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 14:52:10,741::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 14:52:10,743::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 14:52:10,744::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:52:10,745::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:52:10,745::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:52:10,746::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:52:10,746::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:52:10,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:52:11,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:52:11,185::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:52:11,186::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 14:52:11,189::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 14:52:11,191::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 14:52:11,192::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 14:52:11,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:52:11,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:52:11,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:52:11,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00512697 s, 799 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:52:18,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:52:18,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827531 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:52:21,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:52:21,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0231749 s, 177 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:52:28,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:52:28,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138901 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:52:31,742::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:52:31,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00188778 s, 2.2 MB/s\n'; <rc> = 0
>Thread-14293::WARNING::2013-03-17 14:52:36,026::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:52:36,032::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:52:38,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:52:38,347::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881582 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:52:41,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:52:41,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798831 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:52:46,034::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:52:46,035::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:52:46,527::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:52:46,534::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:52:46,535::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:52:48,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:52:48,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012332 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:52:51,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:52:51,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00099527 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:52:58,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:52:58,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166993 s, 245 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:53:01,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:53:01,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765396 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:53:08,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:53:08,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833139 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:53:12,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:53:12,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000927341 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:53:18,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:53:18,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085662 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:53:22,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:53:22,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138863 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:53:28,528::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:53:28,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813007 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:53:32,194::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:53:32,222::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760092 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:53:38,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:53:38,602::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141514 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:53:42,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:53:42,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146805 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14294::WARNING::2013-03-17 14:53:46,563::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:53:46,567::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:53:48,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:53:48,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919898 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:53:52,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:53:52,364::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872861 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:53:56,569::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:53:56,570::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:53:57,168::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:53:57,174::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:53:57,175::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:53:58,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:53:58,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103207 s, 397 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:54:02,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:54:02,456::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864403 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:54:08,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:54:08,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008599 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:54:12,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:54:12,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767015 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:54:18,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:54:18,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017458 s, 235 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:54:22,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:54:22,627::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016604 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:54:28,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:54:28,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00096589 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:54:32,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:54:32,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148176 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:54:38,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:54:38,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167616 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:54:42,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:54:42,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143791 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:54:48,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:54:48,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778635 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:54:52,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:54:52,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001995 s, 2.1 MB/s\n'; <rc> = 0
>Thread-14295::WARNING::2013-03-17 14:54:57,207::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:54:57,213::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:54:58,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:54:58,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172553 s, 237 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:55:02,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:55:02,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000922255 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:55:07,215::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:55:07,216::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:55:07,632::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:55:07,639::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:55:07,640::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:55:08,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:55:09,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011799 s, 347 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:55:12,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:55:13,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00543348 s, 754 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:55:19,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:55:19,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782823 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:55:23,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:55:23,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146628 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:55:29,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:55:29,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000961928 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:55:33,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:55:33,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00071169 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:55:39,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:55:39,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784777 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:55:43,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:55:43,260::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143041 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:55:49,107::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 14:55:49,108::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 14:55:49,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:55:49,163::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0132887 s, 154 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:55:49,164::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0132887 s, 154 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 14:55:49,164::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 14:55:49,165::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 14:55:49,166::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 14:55:49,167::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:55:49,168::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:55:49,168::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:55:49,169::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:55:49,169::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 14:55:49,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:55:49,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:55:49,561::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:55:49,562::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 14:55:49,565::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 14:55:49,566::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 14:55:49,566::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 14:55:49,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:55:49,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:55:49,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:55:49,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134982 s, 303 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:55:53,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:55:53,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146168 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:55:59,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:56:00,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772234 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:56:03,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:56:03,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149376 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14296::WARNING::2013-03-17 14:56:07,668::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:56:07,672::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:56:10,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:56:10,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856935 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:56:13,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:56:13,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112932 s, 3.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:56:17,673::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 14:56:17,675::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:56:17,676::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:56:18,168::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:56:18,175::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:56:18,176::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:56:20,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:56:20,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121813 s, 336 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:56:23,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:56:23,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145556 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:56:30,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:56:30,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176081 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:56:33,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:56:33,671::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134725 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:56:40,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:56:40,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179463 s, 228 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:56:43,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:56:43,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150169 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:56:50,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:56:50,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171611 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:56:53,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:56:53,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126732 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:57:00,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:57:00,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892584 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:57:03,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:57:03,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800077 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:57:10,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:57:10,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850191 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:57:13,970::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 14:57:13,971::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 14:57:13,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:57:14,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00131913 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:57:14,017::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00131913 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 14:57:14,018::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 14:57:14,018::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 14:57:14,020::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 14:57:14,021::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:57:14,022::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:57:14,022::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:57:14,023::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:57:14,023::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 14:57:14,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:57:14,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:57:14,452::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 14:57:14,453::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 14:57:14,456::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 14:57:14,457::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 14:57:14,458::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 14:57:14,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:57:14,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:57:14,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:57:14,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880286 s, 4.7 MB/s\n'; <rc> = 0
>Thread-14297::WARNING::2013-03-17 14:57:18,202::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:57:18,206::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:57:20,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:57:20,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171393 s, 239 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:57:24,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:57:24,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871625 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:57:28,208::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:57:28,209::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:57:28,671::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:57:28,679::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:57:28,681::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:57:30,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:57:30,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129905 s, 315 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:57:34,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:57:34,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891086 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:57:40,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:57:40,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139241 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:57:45,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:57:45,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114171 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:57:50,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:57:50,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000950103 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:57:55,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:57:55,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00655664 s, 625 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:58:00,532::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:58:00,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792709 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:58:05,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:58:05,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015611 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:58:10,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:58:10,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150124 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:58:15,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:58:15,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014469 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:58:20,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:58:20,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141836 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:58:25,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:58:25,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000977436 s, 4.2 MB/s\n'; <rc> = 0
>Thread-14298::WARNING::2013-03-17 14:58:28,711::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:58:28,715::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:58:30,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:58:30,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084551 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:58:35,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:58:35,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178444 s, 2.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:58:38,717::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:58:38,718::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:58:39,180::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:58:39,187::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:58:39,188::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:58:40,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:58:40,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148292 s, 276 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:58:45,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:58:45,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00192858 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:58:50,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:58:50,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00352713 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:58:55,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:58:55,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845143 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:59:00,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:59:00,837::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174179 s, 235 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:59:05,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:59:05,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165012 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:59:10,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:59:10,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137356 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:59:15,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:59:15,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865416 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:59:20,894::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:59:20,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00204446 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:59:25,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:59:25,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141662 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 14:59:30,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:59:30,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159978 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:59:35,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:59:35,950::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774555 s, 5.3 MB/s\n'; <rc> = 0
>Thread-14299::WARNING::2013-03-17 14:59:39,221::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 14:59:39,225::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 14:59:40,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:59:41,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0312018 s, 131 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:59:46,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:59:46,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797847 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 14:59:49,230::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 14:59:49,231::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 14:59:49,657::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 14:59:49,663::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 14:59:49,665::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 14:59:51,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 14:59:51,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106773 s, 384 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 14:59:56,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 14:59:56,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895229 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:00:01,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:00:01,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771394 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:00:06,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:00:06,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000956634 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:00:11,129::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:00:11,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149895 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:00:16,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:00:16,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102975 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:00:21,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:00:21,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167394 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:00:26,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:00:26,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178849 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:00:31,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:00:31,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771973 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:00:36,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:00:36,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00529404 s, 774 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:00:41,254::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:00:41,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847869 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:00:46,429::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:00:46,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00276239 s, 1.5 MB/s\n'; <rc> = 0
>Thread-14300::WARNING::2013-03-17 15:00:49,692::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:00:49,700::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:00:51,294::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 15:00:51,295::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 15:00:51,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:00:51,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000803128 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:00:51,326::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000803128 s, 2.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 15:00:51,327::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 15:00:51,327::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 15:00:51,330::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 15:00:51,331::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:00:51,332::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:00:51,332::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:00:51,333::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:00:51,333::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:00:51,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:00:51,765::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:00:51,769::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:00:51,770::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 15:00:51,773::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 15:00:51,774::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 15:00:51,774::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 15:00:51,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:00:52,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:00:52,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:00:52,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147452 s, 278 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:00:56,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:00:56,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902059 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:00:59,706::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:00:59,707::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:01:00,211::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:01:00,218::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:01:00,219::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:01:02,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:01:02,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166942 s, 245 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:01:06,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:01:06,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148694 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:01:12,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:01:12,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878209 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:01:16,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:01:16,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142293 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:01:22,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:01:22,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000973534 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:01:26,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:01:26,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862842 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:01:32,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:01:32,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834841 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:01:36,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:01:36,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139572 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:01:42,341::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:01:42,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127394 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:01:46,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:01:46,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100111 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:01:52,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:01:52,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185152 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:01:56,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:01:57,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149757 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14301::WARNING::2013-03-17 15:02:00,246::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:02:00,250::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:02:02,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:02:02,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769027 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:02:07,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:02:07,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805167 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:02:10,252::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 15:02:10,253::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:02:10,254::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:02:10,723::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:02:10,730::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:02:10,731::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:02:12,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:02:12,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140066 s, 292 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:02:17,156::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 15:02:17,157::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 15:02:17,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:02:17,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000827841 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:02:17,192::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000827841 s, 2.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 15:02:17,192::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 15:02:17,193::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 15:02:17,194::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 15:02:17,195::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:02:17,196::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:02:17,196::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:02:17,197::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:02:17,197::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:02:17,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:02:17,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:02:17,630::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:02:17,631::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 15:02:17,633::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 15:02:17,635::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 15:02:17,635::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 15:02:17,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:02:18,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:02:18,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:02:18,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176702 s, 232 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:02:22,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:02:22,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152943 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:02:28,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:02:28,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790092 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:02:32,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:02:32,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162929 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:02:38,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:02:38,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821103 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:02:42,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:02:42,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000965097 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:02:48,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:02:48,299::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789591 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:02:52,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:02:52,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173533 s, 236 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:02:58,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:02:58,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088614 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:03:02,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:03:02,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.029649 s, 138 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:03:08,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:03:08,456::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828596 s, 4.9 MB/s\n'; <rc> = 0
>Thread-14302::WARNING::2013-03-17 15:03:10,757::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:03:10,761::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:03:12,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:03:12,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861752 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:03:18,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:03:18,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796425 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:03:20,763::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:03:20,765::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:03:21,204::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:03:21,213::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:03:21,214::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:03:22,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:03:22,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139642 s, 293 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:03:28,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:03:28,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847438 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:03:32,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:03:32,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0213696 s, 192 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:03:38,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:03:38,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149548 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:03:42,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:03:42,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0198166 s, 207 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:03:48,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:03:48,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00309343 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:03:53,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:03:53,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000701995 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:03:58,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:03:58,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179658 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:04:03,033::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:04:03,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741428 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:04:08,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:04:08,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874726 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:04:13,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:04:13,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143852 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:04:19,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:04:19,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000711134 s, 5.8 MB/s\n'; <rc> = 0
>Thread-14303::WARNING::2013-03-17 15:04:21,243::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:04:21,246::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:04:23,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:04:23,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855794 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:04:29,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:04:29,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781636 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:04:31,248::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:04:31,249::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:04:31,670::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:04:31,677::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:04:31,678::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:04:33,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:04:33,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0198258 s, 207 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:04:39,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:04:39,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788812 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:04:43,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:04:43,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151183 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:04:49,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:04:49,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140335 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:04:53,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:04:53,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014044 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:04:59,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:04:59,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00070603 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:05:03,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:05:03,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016001 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:05:09,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:05:09,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147229 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:05:13,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:05:13,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132595 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:05:19,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:05:19,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797174 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:05:23,414::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:05:23,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172372 s, 238 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:05:29,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:05:29,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00187438 s, 2.2 MB/s\n'; <rc> = 0
>Thread-14304::WARNING::2013-03-17 15:05:31,704::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:05:31,708::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:05:33,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:05:33,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826075 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:05:39,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:05:39,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125748 s, 3.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:05:41,710::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:05:41,711::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:05:42,239::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:05:42,246::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:05:42,247::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:05:43,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:05:43,591::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00635805 s, 644 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:05:49,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:05:49,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816639 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:05:53,595::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 15:05:53,597::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 15:05:53,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:05:53,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00119694 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:05:53,639::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00119694 s, 1.7 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 15:05:53,640::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 15:05:53,640::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 15:05:53,642::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 15:05:53,643::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:05:53,643::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:05:53,644::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:05:53,644::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:05:53,644::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:05:53,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:05:54,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:05:54,109::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:05:54,111::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 15:05:54,116::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 15:05:54,117::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 15:05:54,118::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 15:05:54,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:05:54,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:05:54,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:05:54,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00973496 s, 421 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:05:59,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:05:59,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155368 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:06:04,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:06:04,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0257733 s, 159 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:06:09,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:06:09,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013818 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:06:14,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:06:14,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00261542 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:06:20,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:06:20,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805841 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:06:24,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:06:24,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797598 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:06:30,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:06:30,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00544052 s, 753 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:06:34,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:06:34,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174977 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:06:40,178::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:06:40,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000710309 s, 5.8 MB/s\n'; <rc> = 0
>Thread-14305::WARNING::2013-03-17 15:06:42,291::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:06:42,296::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:06:44,792::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:06:44,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158858 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:06:50,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:06:50,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891309 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:06:52,298::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:06:52,299::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:06:52,757::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:06:52,763::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:06:52,765::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:06:54,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:06:54,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115508 s, 355 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:07:00,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:07:00,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000726531 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:07:04,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:07:04,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141304 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:07:10,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:07:10,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864075 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:07:14,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:07:14,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078431 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:07:20,474::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 15:07:20,476::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 15:07:20,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:07:20,511::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00160585 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:07:20,512::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00160585 s, 1.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 15:07:20,513::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 15:07:20,513::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 15:07:20,515::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 15:07:20,516::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:07:20,516::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:07:20,517::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:07:20,517::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:07:20,518::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:07:20,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:07:20,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:07:20,958::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:07:20,959::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 15:07:20,962::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 15:07:20,964::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 15:07:20,964::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 15:07:20,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:07:21,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:07:21,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:07:21,409::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147705 s, 277 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:07:24,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:07:25,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014406 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:07:31,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:07:31,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145653 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:07:35,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:07:35,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149436 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:07:41,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:07:41,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171793 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:07:45,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:07:45,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156128 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:07:51,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:07:51,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000973759 s, 4.2 MB/s\n'; <rc> = 0
>Thread-14306::WARNING::2013-03-17 15:07:52,794::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:07:52,798::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:07:55,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:07:55,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860378 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:08:01,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:08:02,017::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130909 s, 3.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:08:02,800::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 15:08:02,803::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:08:02,804::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:08:03,500::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:08:03,507::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:08:03,508::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:08:05,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:08:05,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117577 s, 348 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:08:12,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:08:12,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829956 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:08:15,209::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:08:15,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148333 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:08:22,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:08:22,163::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864913 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:08:25,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:08:25,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878862 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:08:32,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:08:32,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746885 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:08:35,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:08:35,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083327 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:08:42,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:08:42,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831107 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:08:45,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:08:45,404::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0315677 s, 130 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:08:52,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:08:52,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746158 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:08:55,415::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:08:55,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0213598 s, 192 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:09:02,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:09:02,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00633545 s, 647 kB/s\n'; <rc> = 0
>Thread-14307::WARNING::2013-03-17 15:09:03,536::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:09:03,540::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:09:05,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:09:05,529::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015397 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:09:12,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:09:12,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000986947 s, 4.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:09:13,542::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:09:13,543::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:09:13,978::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:09:13,985::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:09:13,986::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:09:15,537::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:09:15,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135158 s, 303 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:09:22,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:09:22,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016588 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:09:25,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:09:25,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152986 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:09:32,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:09:32,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839294 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:09:35,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:09:35,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143651 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:09:42,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:09:42,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153329 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:09:45,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:09:45,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802857 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:09:52,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:09:53,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00994665 s, 412 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:09:55,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:09:55,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164802 s, 249 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:10:03,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:10:03,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.024968 s, 164 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:10:05,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:10:05,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136114 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:10:13,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:10:13,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152012 s, 269 kB/s\n'; <rc> = 0
>Thread-14308::WARNING::2013-03-17 15:10:14,017::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:10:14,030::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:10:15,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:10:15,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082525 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:10:23,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:10:23,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00427709 s, 958 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:10:24,035::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:10:24,036::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:10:24,541::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:10:24,548::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:10:24,549::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:10:25,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:10:25,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014556 s, 281 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:10:33,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:10:33,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152288 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:10:35,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:10:35,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149239 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:10:43,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:10:43,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792496 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:10:45,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:10:46,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912888 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:10:54,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:10:54,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744422 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:10:56,008::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 15:10:56,010::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 15:10:56,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:10:56,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000854646 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:10:56,046::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000854646 s, 2.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 15:10:56,046::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 15:10:56,046::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 15:10:56,049::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 15:10:56,050::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:10:56,051::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:10:56,051::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:10:56,051::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:10:56,052::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:10:56,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:10:56,491::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:10:56,495::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:10:56,496::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 15:10:56,499::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 15:10:56,500::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 15:10:56,500::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 15:10:56,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:10:56,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:10:56,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:10:56,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118916 s, 344 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:11:04,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:11:04,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155522 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:11:06,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:11:07,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809733 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:11:14,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:11:14,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017919 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:11:17,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:11:17,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000990635 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:11:24,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:11:24,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.012638 s, 324 kB/s\n'; <rc> = 0
>Thread-14309::WARNING::2013-03-17 15:11:24,583::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:11:24,595::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:11:27,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:11:27,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764209 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:11:34,598::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:11:34,600::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:11:34,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:11:34,681::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189207 s, 216 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:11:35,069::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:11:35,076::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:11:35,077::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:11:37,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:11:37,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191779 s, 214 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:11:44,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:11:44,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808049 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:11:47,156::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:11:47,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800468 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:11:54,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:11:54,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000987283 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:11:57,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:11:57,236::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00437983 s, 935 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:12:04,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:12:04,912::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00203138 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:12:07,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:12:07,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00766725 s, 534 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:12:14,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:12:15,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728638 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:12:17,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:12:17,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771733 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:12:25,074::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 15:12:25,076::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 15:12:25,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:12:25,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0007818 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:12:25,114::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0007818 s, 2.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 15:12:25,115::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 15:12:25,115::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 15:12:25,117::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 15:12:25,118::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:12:25,119::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:12:25,120::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:12:25,120::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:12:25,121::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:12:25,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:12:25,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:12:25,545::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:12:25,546::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 15:12:25,549::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 15:12:25,550::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 15:12:25,551::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 15:12:25,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:12:25,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:12:25,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:12:25,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00774697 s, 529 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:12:27,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:12:27,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119702 s, 342 kB/s\n'; <rc> = 0
>Thread-14310::WARNING::2013-03-17 15:12:35,103::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:12:35,109::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 15:12:35,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:12:36,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880693 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:12:37,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:12:37,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078689 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:12:45,119::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:12:45,123::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:12:45,577::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:12:45,583::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:12:45,585::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:12:46,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:12:46,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121073 s, 338 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:12:47,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:12:47,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148928 s, 275 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:12:56,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:12:56,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162007 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:12:57,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:12:57,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018826 s, 218 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:13:06,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:13:06,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778429 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:13:07,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:13:07,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148939 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:13:16,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:13:16,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166064 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:13:17,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:13:17,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017528 s, 234 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:13:26,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:13:26,408::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754415 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:13:27,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:13:27,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152587 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:13:36,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:13:36,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136287 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:13:37,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:13:37,733::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146974 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14311::WARNING::2013-03-17 15:13:45,610::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:13:45,614::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 15:13:46,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:13:46,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770636 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:13:47,738::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:13:47,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159816 s, 256 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:13:55,616::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 15:13:55,617::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:13:55,619::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:13:56,186::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:13:56,193::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:13:56,194::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:13:56,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:13:56,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118463 s, 346 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:13:57,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:13:57,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120208 s, 341 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:14:06,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:14:06,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014547 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:14:07,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:14:07,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153609 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:14:16,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:14:16,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794153 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:14:17,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:14:17,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143688 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:14:26,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:14:26,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843145 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:14:27,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:14:27,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806138 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:14:36,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:14:36,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00231384 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:14:37,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:14:37,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00281796 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:14:47,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:14:47,054::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000956638 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:14:47,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:14:48,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153764 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14312::WARNING::2013-03-17 15:14:56,219::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:14:56,223::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 15:14:57,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:14:57,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161694 s, 253 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:14:58,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:14:58,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762321 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:15:06,225::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:15:06,226::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:15:06,742::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:15:06,750::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:15:06,751::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:15:07,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:15:07,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00895463 s, 457 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:15:08,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:15:08,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00935673 s, 438 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:15:17,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:15:17,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016354 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:15:18,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:15:18,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100527 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:15:27,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:15:27,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172204 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:15:28,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:15:28,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152113 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:15:37,439::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:15:37,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856291 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:15:38,221::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:15:38,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074511 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:15:47,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:15:47,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00258265 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:15:48,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:15:48,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800926 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:15:57,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:15:57,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0209344 s, 196 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:15:58,301::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 15:15:58,302::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 15:15:58,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:15:58,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00161255 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:15:58,350::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00161255 s, 1.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 15:15:58,350::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 15:15:58,351::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 15:15:58,352::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 15:15:58,353::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:15:58,354::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:15:58,354::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:15:58,355::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:15:58,355::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:15:58,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:15:58,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:15:58,776::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:15:58,777::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 15:15:58,779::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 15:15:58,780::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 15:15:58,781::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 15:15:58,782::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:15:59,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:15:59,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:15:59,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138329 s, 296 kB/s\n'; <rc> = 0
>Thread-14313::WARNING::2013-03-17 15:16:06,777::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:16:06,783::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 15:16:07,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:16:07,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862179 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:16:09,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:16:09,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169262 s, 2.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:16:16,785::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:16:16,786::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:16:17,250::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:16:17,257::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:16:17,258::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:16:17,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:16:17,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785193 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:16:19,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:16:19,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193309 s, 212 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:16:27,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:16:27,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728293 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:16:29,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:16:29,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783106 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:16:37,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:16:37,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105599 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:16:39,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:16:39,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104698 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:16:47,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:16:47,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837505 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:16:49,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:16:49,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000944151 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:16:58,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:16:58,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130235 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:16:59,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:16:59,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909192 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:17:08,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:17:08,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000998233 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:17:09,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:17:09,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168906 s, 2.4 MB/s\n'; <rc> = 0
>Thread-14314::WARNING::2013-03-17 15:17:17,287::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:17:17,294::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 15:17:18,221::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:17:18,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000731709 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:17:19,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:17:19,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868992 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:17:27,298::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:17:27,299::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:17:27,834::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:17:27,841::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:17:27,842::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:17:28,292::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 15:17:28,295::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 15:17:28,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:17:28,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0230223 s, 89.0 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:17:28,354::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0230223 s, 89.0 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 15:17:28,355::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 15:17:28,355::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 15:17:28,357::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 15:17:28,358::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:17:28,358::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:17:28,359::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:17:28,359::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:17:28,360::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:17:28,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:17:28,712::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:17:28,717::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:17:28,718::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 15:17:28,720::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 15:17:28,721::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 15:17:28,722::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 15:17:28,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:17:29,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:17:29,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:17:29,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00566269 s, 723 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:17:29,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:17:29,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00343785 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:17:39,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:17:39,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843397 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:17:39,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:17:39,617::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000982745 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:17:49,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:17:49,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821624 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:17:49,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:17:49,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102378 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:17:59,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:17:59,333::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800373 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:17:59,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:17:59,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806343 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:18:09,373::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:18:09,404::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854009 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:18:09,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:18:09,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000987365 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:18:19,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:18:19,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150693 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:18:19,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:18:19,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153743 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14315::WARNING::2013-03-17 15:18:27,878::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:18:27,884::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 15:18:29,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:18:29,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083124 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:18:29,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:18:29,818::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831021 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:18:37,887::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:18:37,889::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:18:38,372::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:18:38,408::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:18:38,409::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:18:39,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:18:39,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143612 s, 285 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:18:39,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:18:39,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118067 s, 347 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:18:49,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:18:49,733::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000691934 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:18:49,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:18:49,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123845 s, 331 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:18:59,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:18:59,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854766 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:18:59,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:18:59,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00728247 s, 562 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:19:09,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:19:09,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000908299 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:19:09,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:19:10,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143387 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:19:19,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:19:19,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0018464 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:19:20,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:19:20,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822609 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:19:30,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:19:30,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:19:30,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153066 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:19:30,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114385 s, 3.6 MB/s\n'; <rc> = 0
>Thread-14316::WARNING::2013-03-17 15:19:38,434::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:19:38,437::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:19:40,129::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:19:40,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:19:40,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790968 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:19:40,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780384 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:19:48,445::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 15:19:48,446::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:19:48,447::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:19:48,893::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:19:48,899::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:19:48,900::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:19:50,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:19:50,254::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:19:50,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010909 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:19:50,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00952078 s, 430 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:20:00,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:20:00,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801236 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:20:00,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:20:00,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167766 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:20:10,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:20:10,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936135 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:20:10,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:20:10,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000973165 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:20:20,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:20:20,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081838 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:20:20,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:20:20,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0251105 s, 163 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:20:30,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:20:30,478::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147165 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:20:30,625::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:20:30,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084254 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:20:40,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:20:40,517::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000708099 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:20:40,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:20:40,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00248189 s, 1.7 MB/s\n'; <rc> = 0
>Thread-14317::WARNING::2013-03-17 15:20:48,929::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:20:48,934::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:20:50,524::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:20:50,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089875 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:20:50,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:20:50,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154612 s, 2.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:20:58,937::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:20:58,938::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:20:59,372::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:20:59,379::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:20:59,380::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:21:00,572::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 15:21:00,575::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 15:21:00,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:21:00,633::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0272912 s, 75.0 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:21:00,634::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0272912 s, 75.0 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 15:21:00,635::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 15:21:00,635::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 15:21:00,637::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 15:21:00,638::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:21:00,638::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:21:00,639::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:21:00,639::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:21:00,640::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:21:00,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:21:00,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:21:00,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155522 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:21:01,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:21:01,082::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:21:01,083::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 15:21:01,085::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 15:21:01,086::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 15:21:01,087::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 15:21:01,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:21:01,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:21:01,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:21:01,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171061 s, 239 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:21:10,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:21:11,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148928 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:21:11,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:21:11,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0245895 s, 167 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:21:21,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:21:21,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790675 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:21:21,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:21:21,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128764 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:21:31,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:21:31,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168323 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:21:31,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:21:31,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819123 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:21:41,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:21:41,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136816 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:21:41,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:21:41,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157132 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:21:51,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:21:51,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178267 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:21:51,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:21:51,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140043 s, 292 kB/s\n'; <rc> = 0
>Thread-14318::WARNING::2013-03-17 15:21:59,410::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:21:59,414::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 15:22:01,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:22:01,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855216 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:22:01,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:22:01,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814259 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:22:09,416::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:22:09,417::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:22:09,868::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:22:09,875::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:22:09,876::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:22:11,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:22:11,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00095626 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:22:11,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:22:11,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140403 s, 292 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:22:21,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:22:21,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00186615 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:22:21,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:22:21,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800873 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:22:31,591::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 15:22:31,592::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 15:22:31,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:22:31,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000774281 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:22:31,621::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000774281 s, 2.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 15:22:31,621::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 15:22:31,622::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 15:22:31,624::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 15:22:31,625::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:22:31,626::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:22:31,626::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:22:31,626::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:22:31,627::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:22:31,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:22:31,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:22:32,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138462 s, 296 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:22:32,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:22:32,057::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:22:32,058::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 15:22:32,060::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 15:22:32,061::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 15:22:32,062::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 15:22:32,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:22:32,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:22:32,445::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:22:32,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00957405 s, 428 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:22:42,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:22:42,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158238 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:22:42,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:22:42,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0210855 s, 194 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:22:52,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:22:52,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163129 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:22:52,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:22:52,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0421099 s, 97.3 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:23:02,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:23:02,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141008 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:23:02,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:23:02,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761176 s, 5.4 MB/s\n'; <rc> = 0
>Thread-14319::WARNING::2013-03-17 15:23:09,904::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:23:09,911::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:23:12,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:23:12,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162627 s, 252 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:23:12,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:23:12,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784767 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:23:19,915::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:23:19,916::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:23:20,344::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:23:20,351::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:23:20,353::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:23:22,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:23:22,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168578 s, 243 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:23:22,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:23:22,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165877 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:23:32,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:23:32,311::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204817 s, 200 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:23:32,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:23:33,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001663 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:23:42,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:23:42,364::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164855 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:23:43,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:23:43,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000941248 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:23:52,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:23:52,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016621 s, 246 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:23:53,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:23:53,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142992 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:24:02,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:24:02,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142437 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:24:03,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:24:03,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148078 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:24:12,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:24:12,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856389 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:24:13,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:24:13,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0198617 s, 206 kB/s\n'; <rc> = 0
>Thread-14320::WARNING::2013-03-17 15:24:20,380::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:24:20,386::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:24:22,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:24:22,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825337 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:24:23,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:24:23,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00370432 s, 1.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:24:30,393::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:24:30,395::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:24:30,856::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:24:30,863::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:24:30,864::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:24:32,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:24:32,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019821 s, 207 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:24:33,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:24:33,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0018522 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:24:42,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:24:42,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013933 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:24:43,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:24:43,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00215584 s, 1.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:24:52,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:24:52,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00285784 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:24:53,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:24:53,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000730194 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:25:02,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:25:02,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133811 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:25:03,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:25:03,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148839 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:25:12,814::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:25:12,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146638 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:25:13,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:25:13,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156779 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:25:22,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:25:22,899::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836667 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:25:23,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:25:23,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000951553 s, 4.3 MB/s\n'; <rc> = 0
>Thread-14321::WARNING::2013-03-17 15:25:30,903::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:25:30,907::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:25:32,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:25:32,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139746 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:25:33,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:25:33,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000703569 s, 5.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:25:40,908::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 15:25:40,910::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:25:40,911::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:25:41,335::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:25:41,342::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:25:41,343::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:25:42,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:25:42,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166661 s, 246 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:25:44,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:25:44,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124374 s, 329 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:25:52,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:25:53,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00245518 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:25:54,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:25:54,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847723 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:26:03,026::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 15:26:03,027::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 15:26:03,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:26:03,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000812395 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:26:03,055::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000812395 s, 2.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 15:26:03,055::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 15:26:03,055::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 15:26:03,057::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 15:26:03,058::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:26:03,058::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:26:03,059::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:26:03,059::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:26:03,060::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:26:03,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:26:03,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:26:03,478::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:26:03,479::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 15:26:03,481::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 15:26:03,482::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 15:26:03,483::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 15:26:03,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:26:03,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:26:03,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:26:03,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00500484 s, 818 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:26:04,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:26:04,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180521 s, 227 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:26:13,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:26:13,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00686261 s, 597 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:26:14,292::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:26:14,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017441 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:26:23,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:26:24,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0259625 s, 158 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:26:24,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:26:24,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144348 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:26:34,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:26:34,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180023 s, 228 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:26:34,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:26:34,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741676 s, 5.5 MB/s\n'; <rc> = 0
>Thread-14322::WARNING::2013-03-17 15:26:41,370::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:26:41,374::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:26:44,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:26:44,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0268847 s, 152 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:26:44,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:26:44,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802194 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:26:51,376::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:26:51,377::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:26:51,839::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:26:51,846::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:26:51,847::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:26:54,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:26:54,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105848 s, 387 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:26:54,617::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:26:54,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784083 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:27:04,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:27:04,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00688435 s, 595 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:27:04,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:27:04,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849719 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:27:14,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:27:14,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173047 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:27:14,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:27:14,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0298876 s, 137 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:27:24,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:27:24,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00764645 s, 536 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:27:24,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:27:24,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0242867 s, 169 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:27:34,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:27:34,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170251 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:27:34,964::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 15:27:34,965::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 15:27:34,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:27:35,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00133834 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:27:35,007::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00133834 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 15:27:35,009::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 15:27:35,010::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 15:27:35,011::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 15:27:35,012::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:27:35,013::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:27:35,013::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:27:35,013::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:27:35,014::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:27:35,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:27:35,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:27:35,443::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:27:35,444::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 15:27:35,446::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 15:27:35,448::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 15:27:35,448::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 15:27:35,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:27:35,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:27:35,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:27:35,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135368 s, 303 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:27:44,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:27:44,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134772 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:27:45,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:27:46,010::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00806969 s, 508 kB/s\n'; <rc> = 0
>Thread-14323::WARNING::2013-03-17 15:27:51,891::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:27:51,894::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:27:54,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:27:54,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781859 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:27:56,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:27:56,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000937377 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:28:01,895::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:28:01,897::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:28:02,500::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:28:02,507::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:28:02,508::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:28:04,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:28:04,511::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154515 s, 265 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:28:06,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:28:06,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127359 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:28:14,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:28:14,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150694 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:28:16,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:28:16,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160317 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:28:24,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:28:24,602::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894334 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:28:26,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:28:26,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206641 s, 198 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:28:34,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:28:34,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140618 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:28:36,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:28:36,408::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00868452 s, 472 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:28:44,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:28:44,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139215 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:28:46,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:28:46,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0219973 s, 186 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:28:54,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:28:54,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865781 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:28:56,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:28:56,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172687 s, 2.4 MB/s\n'; <rc> = 0
>Thread-14324::WARNING::2013-03-17 15:29:02,537::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:29:02,541::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:29:04,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:29:04,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156123 s, 262 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:29:06,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:29:06,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156173 s, 2.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:29:12,543::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:29:12,545::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:29:12,962::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:29:12,969::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:29:12,970::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:29:14,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:29:14,890::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00438787 s, 933 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:29:16,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:29:16,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179079 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:29:24,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:29:24,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175917 s, 233 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:29:26,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:29:26,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792907 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:29:34,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:29:34,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154612 s, 265 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:29:36,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:29:36,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00190923 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:29:45,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:29:45,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087753 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:29:46,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:29:46,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149502 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:29:55,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:29:55,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145386 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:29:57,014::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:29:57,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808274 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:30:05,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:30:05,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145054 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:30:07,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:30:07,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116447 s, 3.5 MB/s\n'; <rc> = 0
>Thread-14325::WARNING::2013-03-17 15:30:13,000::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:30:13,006::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:30:15,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:30:15,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882202 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:30:17,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:30:17,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828991 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:30:23,009::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:30:23,010::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:30:23,422::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:30:23,429::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:30:23,431::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:30:25,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:30:25,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115994 s, 353 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:30:27,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:30:27,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134829 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:30:35,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:30:35,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000983539 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:30:37,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:30:37,372::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.021427 s, 191 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:30:45,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:30:45,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001315 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:30:47,412::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:30:47,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0223522 s, 183 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:30:55,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:30:55,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171793 s, 238 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:30:57,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:30:57,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00096806 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:31:05,347::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 15:31:05,348::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 15:31:05,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:31:05,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0014421 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:31:05,396::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0014421 s, 1.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 15:31:05,396::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 15:31:05,396::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 15:31:05,398::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 15:31:05,399::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:31:05,400::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:31:05,400::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:31:05,400::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:31:05,401::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:31:05,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:31:05,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:31:05,836::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:31:05,837::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 15:31:05,839::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 15:31:05,841::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 15:31:05,841::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 15:31:05,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:31:06,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:31:06,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:31:06,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187785 s, 218 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:31:07,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:31:07,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0236228 s, 173 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:31:16,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:31:16,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014397 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:31:17,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:31:17,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0256777 s, 160 kB/s\n'; <rc> = 0
>Thread-14326::WARNING::2013-03-17 15:31:23,468::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:31:23,471::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:31:26,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:31:26,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175143 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:31:27,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:31:27,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00193405 s, 2.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:31:33,474::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 15:31:33,475::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:31:33,476::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:31:33,893::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:31:33,900::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:31:33,901::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:31:36,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:31:36,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159057 s, 258 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:31:37,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:31:37,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851366 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:31:46,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:31:46,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901902 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:31:47,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:31:48,010::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0215108 s, 190 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:31:56,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:31:56,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846969 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:31:58,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:31:58,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897214 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:32:06,492::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:32:06,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146647 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:32:08,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:32:08,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154515 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:32:16,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:32:16,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013996 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:32:18,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:32:18,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846069 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:32:26,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:32:26,636::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016417 s, 249 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:32:28,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:32:28,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156984 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14327::WARNING::2013-03-17 15:32:33,932::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:32:33,935::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:32:36,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:32:36,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837949 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:32:38,364::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 15:32:38,366::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 15:32:38,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:32:38,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00152783 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:32:38,401::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00152783 s, 1.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 15:32:38,402::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 15:32:38,402::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 15:32:38,404::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 15:32:38,405::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:32:38,406::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:32:38,406::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:32:38,406::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:32:38,407::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:32:38,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:32:38,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:32:38,841::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:32:38,842::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 15:32:38,844::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 15:32:38,845::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 15:32:38,846::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 15:32:38,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:32:39,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:32:39,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:32:39,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00841416 s, 487 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:32:43,937::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:32:43,938::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:32:44,401::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:32:44,408::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:32:44,410::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:32:46,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:32:46,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107969 s, 379 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:32:49,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:32:49,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112876 s, 363 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:32:56,732::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:32:56,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125375 s, 327 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:32:59,352::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:32:59,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083484 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:33:06,778::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:33:06,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185472 s, 221 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:33:09,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:33:09,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175045 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:33:16,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:33:16,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144403 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:33:19,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:33:19,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000742329 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:33:26,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:33:26,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146157 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:33:29,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:33:29,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828672 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:33:36,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:33:36,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110391 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:33:39,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:33:39,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0229839 s, 178 kB/s\n'; <rc> = 0
>Thread-14328::WARNING::2013-03-17 15:33:44,437::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:33:44,441::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:33:46,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:33:47,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862231 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:33:49,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:33:49,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113253 s, 362 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:33:54,443::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:33:54,444::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:33:54,883::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:33:54,890::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:33:54,892::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:33:57,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:33:57,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0070663 s, 580 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:33:59,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:33:59,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142719 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:34:07,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:34:07,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867662 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:34:09,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:34:09,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834528 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:34:17,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:34:17,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00338809 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:34:20,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:34:20,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154287 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:34:27,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:34:27,175::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0157343 s, 260 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:34:30,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:34:30,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862389 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:34:37,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:34:37,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852605 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:34:40,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:34:40,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160763 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:34:47,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:34:47,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133972 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:34:50,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:34:50,297::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152652 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14329::WARNING::2013-03-17 15:34:54,925::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:34:54,927::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:34:57,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:34:57,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775867 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:35:00,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:35:00,364::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079873 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:35:04,929::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:35:04,931::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:35:05,348::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:35:05,355::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:35:05,356::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:35:07,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:35:07,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00964986 s, 424 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:35:10,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:35:10,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799792 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:35:17,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:35:17,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145387 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:35:20,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:35:20,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0213683 s, 192 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:35:27,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:35:27,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00720461 s, 569 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:35:30,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:35:30,593::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000713717 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:35:37,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:35:37,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897893 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:35:40,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:35:40,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846962 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:35:47,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:35:47,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143311 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:35:50,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:35:50,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203317 s, 201 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:35:57,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:35:57,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077589 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:36:00,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:36:00,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124084 s, 3.3 MB/s\n'; <rc> = 0
>Thread-14330::WARNING::2013-03-17 15:36:05,386::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:36:05,390::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:36:07,545::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 15:36:07,546::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 15:36:07,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:36:07,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000855886 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:36:07,576::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000855886 s, 2.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 15:36:07,577::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 15:36:07,577::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 15:36:07,579::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 15:36:07,581::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:36:07,582::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:36:07,582::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:36:07,583::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:36:07,583::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:36:07,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:36:08,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:36:08,018::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:36:08,019::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 15:36:08,021::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 15:36:08,023::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 15:36:08,023::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 15:36:08,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:36:08,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:36:08,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:36:08,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160843 s, 255 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:36:10,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:36:10,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822622 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:36:15,392::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:36:15,393::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:36:15,856::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:36:15,863::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:36:15,864::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:36:18,439::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:36:18,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00997645 s, 411 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:36:20,959::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:36:21,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0237864 s, 172 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:36:28,490::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:36:28,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015567 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:36:31,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:36:31,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143572 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:36:38,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:36:38,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100613 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:36:41,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:36:41,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133518 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:36:48,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:36:48,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165685 s, 247 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:36:51,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:36:51,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805433 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:36:58,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:36:58,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799695 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:37:01,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:37:01,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142115 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:37:08,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:37:08,698::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801592 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:37:11,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:37:11,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136916 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14331::WARNING::2013-03-17 15:37:15,892::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:37:15,898::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:37:18,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:37:18,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132189 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:37:21,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:37:21,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891303 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:37:25,899::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 15:37:25,900::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:37:25,901::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:37:26,348::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:37:26,355::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:37:26,358::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:37:28,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:37:28,793::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114015 s, 359 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:37:31,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:37:31,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839176 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:37:38,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:37:38,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141416 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:37:41,579::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 15:37:41,580::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 15:37:41,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:37:41,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0243492 s, 84.1 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:37:41,631::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0243492 s, 84.1 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 15:37:41,632::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 15:37:41,632::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 15:37:41,634::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 15:37:41,635::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:37:41,636::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:37:41,636::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:37:41,636::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:37:41,637::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:37:41,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:37:42,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:37:42,055::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:37:42,056::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 15:37:42,059::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 15:37:42,060::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 15:37:42,061::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 15:37:42,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:37:42,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:37:42,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:37:42,450::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911038 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:37:48,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:37:48,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839755 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:37:52,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:37:52,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000944462 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:37:58,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:37:58,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00098121 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:38:02,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:38:02,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146244 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:38:08,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:38:08,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165961 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:38:12,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:38:12,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837745 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:38:18,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:38:19,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136432 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:38:22,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:38:22,766::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152699 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14332::WARNING::2013-03-17 15:38:26,390::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:38:26,392::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:38:29,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:38:29,046::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771772 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:38:32,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:38:32,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101249 s, 4.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:38:36,394::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:38:36,396::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:38:36,899::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:38:36,906::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:38:36,907::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:38:39,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:38:39,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179184 s, 229 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:38:42,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:38:42,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0262763 s, 156 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:38:49,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:38:49,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808022 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:38:52,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:38:53,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102438 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:38:59,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:38:59,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814846 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:39:03,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:39:03,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871112 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:39:09,171::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:39:09,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807044 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:39:13,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:39:13,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166069 s, 247 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:39:19,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:39:19,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013669 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:39:23,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:39:23,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761427 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:39:29,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:39:29,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014104 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:39:33,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:39:33,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0218678 s, 187 kB/s\n'; <rc> = 0
>Thread-14333::WARNING::2013-03-17 15:39:36,937::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:39:36,946::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:39:39,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:39:39,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000978263 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:39:43,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:39:43,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859631 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:39:46,949::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:39:46,950::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:39:47,391::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:39:47,398::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:39:47,399::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:39:49,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:39:49,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161586 s, 253 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:39:53,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:39:53,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085475 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:39:59,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:39:59,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089168 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:40:03,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:40:03,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0205815 s, 199 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:40:09,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:40:09,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000725769 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:40:13,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:40:13,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851399 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:40:19,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:40:19,484::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843451 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:40:23,685::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:40:23,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177488 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:40:29,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:40:29,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152584 s, 268 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:40:33,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:40:33,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001605 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:40:39,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:40:39,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747402 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:40:43,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:40:43,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117923 s, 3.5 MB/s\n'; <rc> = 0
>Thread-14334::WARNING::2013-03-17 15:40:47,425::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:40:47,428::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:40:49,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:40:49,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081682 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:40:53,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:40:53,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0205992 s, 199 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:40:57,441::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:40:57,442::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:40:57,886::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:40:57,893::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:40:57,894::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:40:59,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:40:59,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101202 s, 405 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:41:04,014::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:41:04,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100458 s, 408 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:41:09,670::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 15:41:09,671::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 15:41:09,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:41:09,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00183996 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:41:09,716::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00183996 s, 1.1 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 15:41:09,716::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 15:41:09,717::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 15:41:09,718::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 15:41:09,719::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:41:09,720::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:41:09,720::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:41:09,720::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:41:09,721::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:41:09,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:41:10,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:41:10,175::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:41:10,175::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 15:41:10,178::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 15:41:10,179::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 15:41:10,179::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 15:41:10,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:41:10,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:41:10,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:41:10,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155017 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:41:14,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:41:14,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000929314 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:41:20,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:41:20,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149806 s, 273 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:41:24,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:41:24,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130266 s, 314 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:41:30,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:41:30,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897933 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:41:34,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:41:34,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772167 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:41:40,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:41:40,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139667 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:41:44,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:41:44,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114476 s, 358 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:41:50,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:41:50,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741819 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:41:54,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:41:54,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152422 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14335::WARNING::2013-03-17 15:41:57,919::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:41:57,923::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:42:00,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:42:00,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00278936 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:42:04,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:42:04,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920226 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:42:07,926::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:42:07,927::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:42:08,356::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:42:08,363::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:42:08,364::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:42:10,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:42:10,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00916066 s, 447 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:42:14,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:42:14,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861082 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:42:20,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:42:20,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076907 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:42:24,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:42:24,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00267998 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:42:30,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:42:30,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164114 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:42:34,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:42:34,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142203 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:42:40,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:42:40,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774012 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:42:44,789::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 15:42:44,790::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 15:42:44,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:42:44,819::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000759147 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:42:44,820::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000759147 s, 2.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 15:42:44,820::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 15:42:44,821::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 15:42:44,822::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 15:42:44,823::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:42:44,824::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:42:44,825::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:42:44,825::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:42:44,825::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:42:44,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:42:45,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:42:45,251::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:42:45,252::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 15:42:45,255::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 15:42:45,256::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 15:42:45,256::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 15:42:45,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:42:45,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:42:45,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:42:45,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00666113 s, 615 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:42:50,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:42:51,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150233 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:42:55,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:42:55,818::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0211739 s, 193 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:43:01,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:43:01,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00372119 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:43:05,857::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:43:05,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887379 s, 4.6 MB/s\n'; <rc> = 0
>Thread-14336::WARNING::2013-03-17 15:43:08,391::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:43:08,398::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:43:11,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:43:11,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811393 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:43:15,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:43:15,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783003 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:43:18,400::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 15:43:18,401::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:43:18,402::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:43:18,926::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:43:18,932::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:43:18,934::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:43:21,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:43:21,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00871553 s, 470 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:43:26,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:43:26,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110578 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:43:31,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:43:31,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159484 s, 257 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:43:36,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:43:36,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877622 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:43:41,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:43:41,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079071 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:43:46,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:43:46,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170359 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:43:51,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:43:51,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142679 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:43:56,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:43:56,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147921 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:44:01,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:44:01,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109052 s, 376 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:44:06,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:44:06,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137925 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:44:11,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:44:11,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148393 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:44:16,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:44:16,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842986 s, 4.9 MB/s\n'; <rc> = 0
>Thread-14337::WARNING::2013-03-17 15:44:18,961::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:44:18,964::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:44:21,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:44:21,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769879 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:44:26,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:44:26,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825464 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:44:28,970::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:44:28,971::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:44:29,389::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:44:29,396::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:44:29,398::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:44:31,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:44:31,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107683 s, 380 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:44:36,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:44:36,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000736025 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:44:41,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:44:41,548::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814129 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:44:46,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:44:46,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138017 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:44:51,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:44:51,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873548 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:44:56,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:44:56,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782756 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:45:01,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:45:01,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152396 s, 269 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:45:06,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:45:06,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820004 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:45:11,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:45:11,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163313 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:45:16,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:45:16,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769857 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:45:21,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:45:21,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151106 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:45:26,933::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:45:26,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825883 s, 5.0 MB/s\n'; <rc> = 0
>Thread-14338::WARNING::2013-03-17 15:45:29,431::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:45:29,438::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:45:31,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:45:31,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842761 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:45:37,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:45:37,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904425 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:45:39,441::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:45:39,443::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:45:39,918::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:45:39,925::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:45:39,926::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:45:41,782::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:45:41,819::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00899894 s, 455 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:45:47,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:45:47,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849788 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:45:51,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:45:51,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000719788 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:45:57,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:45:57,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199404 s, 205 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:46:01,863::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:46:01,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125623 s, 326 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:46:07,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:46:07,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756526 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:46:11,937::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 15:46:11,938::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 15:46:11,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:46:11,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000821466 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:46:11,991::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000821466 s, 2.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 15:46:11,993::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 15:46:11,995::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 15:46:12,001::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 15:46:12,010::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:46:12,011::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:46:12,011::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:46:12,012::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:46:12,012::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:46:12,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:46:12,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:46:12,440::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:46:12,441::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 15:46:12,445::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 15:46:12,446::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 15:46:12,447::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 15:46:12,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:46:12,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:46:12,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:46:12,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144453 s, 284 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:46:17,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:46:17,398::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178104 s, 230 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:46:22,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:46:22,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126179 s, 325 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:46:27,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:46:27,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0236241 s, 173 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:46:32,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:46:32,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740353 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:46:37,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:46:37,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111859 s, 3.7 MB/s\n'; <rc> = 0
>Thread-14339::WARNING::2013-03-17 15:46:39,952::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:46:39,957::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:46:42,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:46:43,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173314 s, 236 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:46:47,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:46:47,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756618 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:46:49,961::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:46:49,962::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:46:50,417::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:46:50,425::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:46:50,426::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:46:53,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:46:53,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0081718 s, 501 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:46:57,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:46:57,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877562 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:47:03,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:47:03,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00218156 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:47:07,778::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:47:07,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0201715 s, 203 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:47:13,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:47:13,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00780881 s, 525 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:47:17,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:47:17,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0202042 s, 203 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:47:23,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:47:23,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156184 s, 262 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:47:27,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:47:27,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748255 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:47:33,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:47:33,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771927 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:47:38,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:47:38,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159584 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:47:43,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:47:43,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161206 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:47:48,103::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 15:47:48,104::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 15:47:48,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:47:48,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00129239 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:47:48,160::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00129239 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 15:47:48,162::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 15:47:48,162::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 15:47:48,164::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 15:47:48,165::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:47:48,167::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:47:48,168::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:47:48,168::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:47:48,169::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:47:48,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:47:48,602::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:47:48,606::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:47:48,607::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 15:47:48,610::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 15:47:48,611::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 15:47:48,611::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 15:47:48,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:47:49,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:47:49,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:47:49,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142428 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14340::WARNING::2013-03-17 15:47:50,454::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:47:50,462::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:47:53,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:47:53,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780108 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:47:59,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:47:59,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780634 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:48:00,465::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:48:00,466::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:48:01,013::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:48:01,019::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:48:01,020::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:48:03,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:48:03,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00898403 s, 456 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:48:09,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:48:09,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00386557 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:48:13,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:48:13,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894615 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:48:19,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:48:19,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000991977 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:48:23,412::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:48:23,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000713259 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:48:29,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:48:29,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0214909 s, 191 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:48:33,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:48:33,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868827 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:48:39,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:48:39,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00349788 s, 1.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:48:43,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:48:43,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161429 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:48:49,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:48:49,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808173 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:48:53,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:48:53,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141185 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:48:59,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:48:59,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082157 s, 5.0 MB/s\n'; <rc> = 0
>Thread-14341::WARNING::2013-03-17 15:49:01,048::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:49:01,052::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:49:03,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:49:03,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080023 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:49:09,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:49:09,658::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106234 s, 3.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:49:11,054::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 15:49:11,055::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:49:11,056::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:49:11,551::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:49:11,558::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:49:11,560::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:49:13,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:49:13,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127886 s, 320 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:49:19,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:49:19,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159442 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:49:23,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:49:23,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833009 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:49:29,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:49:29,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173442 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:49:33,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:49:33,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014021 s, 292 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:49:39,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:49:39,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842042 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:49:43,760::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:49:43,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168209 s, 244 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:49:49,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:49:49,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000940214 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:49:53,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:49:53,853::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134967 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:50:00,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:50:00,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016337 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:50:03,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:50:03,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014872 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:50:10,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:50:10,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107875 s, 3.8 MB/s\n'; <rc> = 0
>Thread-14342::WARNING::2013-03-17 15:50:11,589::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:50:11,594::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:50:13,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:50:13,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758479 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:50:20,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:50:20,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835677 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:50:21,597::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:50:21,600::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:50:22,106::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:50:22,113::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:50:22,114::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:50:23,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:50:23,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00723274 s, 566 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:50:30,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:50:30,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824697 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:50:33,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:50:34,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147326 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:50:40,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:50:40,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008891 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:50:44,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:50:44,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149116 s, 275 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:50:50,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:50:50,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014483 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:50:54,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:50:54,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114605 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:51:00,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:51:00,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813821 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:51:04,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:51:04,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133385 s, 307 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:51:10,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:51:10,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131003 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:51:14,188::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 15:51:14,193::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 15:51:14,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:51:14,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000961215 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:51:14,226::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000961215 s, 2.1 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 15:51:14,227::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 15:51:14,227::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 15:51:14,228::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 15:51:14,229::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:51:14,230::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:51:14,230::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:51:14,231::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:51:14,231::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:51:14,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:51:14,658::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:51:14,662::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:51:14,663::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 15:51:14,665::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 15:51:14,667::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 15:51:14,667::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 15:51:14,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:51:15,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:51:15,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:51:15,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105487 s, 388 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:51:20,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:51:20,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109411 s, 3.7 MB/s\n'; <rc> = 0
>Thread-14343::WARNING::2013-03-17 15:51:22,142::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:51:22,148::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:51:25,128::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:51:25,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817779 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:51:30,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:51:30,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0220702 s, 186 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:51:32,150::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:51:32,151::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:51:32,676::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:51:32,683::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:51:32,684::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:51:35,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:51:35,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137854 s, 297 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:51:40,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:51:40,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898554 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:51:45,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:51:45,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761592 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:51:50,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:51:50,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0232054 s, 177 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:51:55,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:51:55,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000949085 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:52:01,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:52:01,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809913 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:52:05,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:52:05,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085607 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:52:11,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:52:11,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755967 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:52:15,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:52:15,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902544 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:52:21,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:52:21,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103896 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:52:25,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:52:25,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082551 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:52:31,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:52:31,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100241 s, 4.1 MB/s\n'; <rc> = 0
>Thread-14344::WARNING::2013-03-17 15:52:32,711::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:52:32,718::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:52:35,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:52:35,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842363 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:52:41,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:52:41,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867386 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:52:42,726::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:52:42,727::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:52:43,228::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:52:43,235::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:52:43,236::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:52:45,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:52:45,483::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116523 s, 352 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:52:51,394::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 15:52:51,395::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 15:52:51,396::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:52:51,424::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000716174 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:52:51,426::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000716174 s, 2.9 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 15:52:51,426::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 15:52:51,426::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 15:52:51,428::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 15:52:51,429::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:52:51,430::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:52:51,430::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:52:51,431::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:52:51,431::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:52:51,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:52:51,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:52:51,880::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:52:51,881::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 15:52:51,884::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 15:52:51,885::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 15:52:51,886::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 15:52:51,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:52:52,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:52:52,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:52:52,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010814 s, 379 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:52:55,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:52:55,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146002 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:53:02,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:53:02,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806112 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:53:05,537::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:53:05,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148779 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:53:12,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:53:12,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143573 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:53:15,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:53:15,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074204 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:53:22,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:53:22,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169602 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:53:25,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:53:25,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795494 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:53:32,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:53:32,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156063 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:53:35,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:53:35,692::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151547 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:53:42,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:53:42,837::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000963474 s, 4.3 MB/s\n'; <rc> = 0
>Thread-14345::WARNING::2013-03-17 15:53:43,282::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:53:43,285::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:53:45,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:53:45,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008162 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:53:52,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:53:53,010::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00903833 s, 453 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:53:53,288::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:53:53,290::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:53:53,754::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:53:53,760::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:53:53,762::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:53:55,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:53:55,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147406 s, 278 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:54:03,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:54:03,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811541 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:54:05,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:54:05,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824192 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:54:13,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:54:13,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136825 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:54:15,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:54:15,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756994 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:54:23,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:54:23,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763487 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:54:25,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:54:25,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00256271 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:54:33,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:54:33,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0196682 s, 208 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:54:35,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:54:35,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148526 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:54:43,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:54:43,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088991 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:54:45,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:54:46,010::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170761 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:54:53,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:54:53,529::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816448 s, 5.0 MB/s\n'; <rc> = 0
>Thread-14346::WARNING::2013-03-17 15:54:53,792::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:54:53,799::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 15:54:56,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:54:56,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157606 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:55:03,782::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:55:03,808::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 15:55:03,829::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:55:03,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000905664 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:55:03,844::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:55:04,501::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:55:04,507::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:55:04,508::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:55:06,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:55:06,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108352 s, 378 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:55:14,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:55:14,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177823 s, 230 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:55:16,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:55:16,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129865 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:55:24,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:55:24,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108547 s, 377 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:55:26,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:55:26,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00095012 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:55:34,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:55:34,358::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142271 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:55:36,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:55:36,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149916 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:55:44,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:55:44,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0295124 s, 139 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:55:46,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:55:46,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178231 s, 230 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:55:54,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:55:54,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168204 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:55:56,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:55:56,350::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887483 s, 4.6 MB/s\n'; <rc> = 0
>Thread-14347::WARNING::2013-03-17 15:56:04,544::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:56:04,548::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 15:56:04,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:56:04,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0245026 s, 167 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:56:06,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:56:06,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168585 s, 2.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:56:14,550::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:56:14,551::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:56:14,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:56:14,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150891 s, 271 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:56:15,066::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:56:15,072::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:56:15,073::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:56:16,402::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 15:56:16,405::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 15:56:16,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:56:16,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.012551 s, 163 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:56:16,448::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.012551 s, 163 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 15:56:16,449::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 15:56:16,449::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 15:56:16,451::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 15:56:16,453::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:56:16,454::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:56:16,454::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:56:16,455::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:56:16,455::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 15:56:16,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:56:16,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:56:16,874::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:56:16,875::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 15:56:16,879::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 15:56:16,880::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 15:56:16,881::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 15:56:16,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:56:17,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:56:17,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:56:17,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00634369 s, 646 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:56:24,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:56:24,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000735508 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:56:27,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:56:27,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117255 s, 349 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:56:34,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:56:34,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145986 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:56:37,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:56:37,347::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122787 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:56:44,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:56:44,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152034 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:56:47,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:56:47,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799357 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:56:55,033::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:56:55,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877815 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:56:57,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:56:57,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142082 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:57:05,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:57:05,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810071 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:57:07,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:57:07,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151744 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14348::WARNING::2013-03-17 15:57:15,100::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:57:15,104::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 15:57:15,171::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:57:15,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158179 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:57:17,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:57:17,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00390112 s, 1.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:57:25,106::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:57:25,107::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:57:25,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:57:25,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156411 s, 262 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:57:25,596::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:57:25,601::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:57:25,603::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 15:57:27,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:57:27,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193758 s, 211 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:57:35,365::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:57:35,395::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000976194 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:57:37,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:57:37,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158157 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:57:45,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:57:45,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885271 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:57:47,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:57:47,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143218 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:57:55,503::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 15:57:55,504::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 15:57:55,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:57:55,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0213646 s, 95.9 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:57:55,560::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0213646 s, 95.9 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 15:57:55,560::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 15:57:55,560::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 15:57:55,563::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 15:57:55,564::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:57:55,565::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:57:55,565::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:57:55,565::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:57:55,566::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 15:57:55,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:57:55,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:57:55,969::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:57:55,970::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 15:57:55,972::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 15:57:55,974::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 15:57:55,974::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 15:57:55,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:57:56,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:57:56,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:57:56,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109294 s, 375 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:57:57,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:57:57,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00472033 s, 868 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:58:06,455::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:58:06,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143609 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:58:07,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:58:07,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791689 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:58:16,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:58:16,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134729 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:58:17,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:58:17,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165698 s, 247 kB/s\n'; <rc> = 0
>Thread-14349::WARNING::2013-03-17 15:58:25,628::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:58:25,634::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 15:58:26,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:58:26,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008881 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:58:27,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:58:27,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804135 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:58:35,636::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:58:35,637::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:58:36,056::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:58:36,063::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:58:36,065::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:58:36,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:58:36,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126336 s, 324 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:58:37,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:58:37,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100288 s, 408 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:58:46,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:58:46,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0237226 s, 173 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:58:47,899::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:58:47,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902921 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:58:56,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:58:56,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00090363 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:58:57,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:58:57,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00242466 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:59:06,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:59:07,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144532 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:59:07,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:59:08,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00262259 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:59:17,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:59:17,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160683 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:59:18,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:59:18,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00289022 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:59:27,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:59:27,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165538 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:59:28,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:59:28,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137669 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14350::WARNING::2013-03-17 15:59:36,092::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 15:59:36,097::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 15:59:37,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:59:37,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0219817 s, 186 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:59:38,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:59:38,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173336 s, 2.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 15:59:46,103::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 15:59:46,104::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 15:59:46,589::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 15:59:46,596::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 15:59:46,597::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 15:59:47,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:59:47,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00707999 s, 579 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:59:48,171::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:59:48,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131741 s, 311 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 15:59:57,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 15:59:57,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000700241 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 15:59:58,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 15:59:58,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164118 s, 250 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:00:07,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:00:07,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00533322 s, 768 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:00:08,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:00:08,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886051 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:00:17,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:00:17,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00943818 s, 434 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:00:18,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:00:18,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165848 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:00:27,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:00:27,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156405 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:00:28,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:00:28,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855329 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:00:37,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:00:37,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0226827 s, 181 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:00:38,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:00:38,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780146 s, 5.3 MB/s\n'; <rc> = 0
>Thread-14351::WARNING::2013-03-17 16:00:46,630::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:00:46,634::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 16:00:47,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:00:47,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847965 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:00:48,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:00:48,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0200006 s, 205 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:00:56,636::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 16:00:56,637::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:00:56,638::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:00:57,063::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:00:57,070::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:00:57,071::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:00:57,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:00:57,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206182 s, 199 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:00:58,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:00:58,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00968302 s, 423 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:01:07,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:01:08,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000695408 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:01:08,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:01:08,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829881 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:01:18,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:01:18,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155898 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:01:18,583::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 16:01:18,587::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 16:01:18,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:01:18,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0154727 s, 132 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:01:18,635::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0154727 s, 132 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 16:01:18,636::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 16:01:18,636::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 16:01:18,638::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 16:01:18,639::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:01:18,639::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:01:18,640::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:01:18,640::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:01:18,640::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:01:18,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:01:19,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:01:19,125::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:01:19,126::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 16:01:19,128::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 16:01:19,130::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 16:01:19,130::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 16:01:19,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:01:19,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:01:19,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:01:19,521::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00646873 s, 633 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:01:28,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:01:28,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148444 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:01:29,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:01:29,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168225 s, 243 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:01:38,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:01:38,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169095 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:01:39,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:01:39,636::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015795 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:01:48,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:01:48,350::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128717 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:01:49,647::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:01:49,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00225913 s, 1.8 MB/s\n'; <rc> = 0
>Thread-14352::WARNING::2013-03-17 16:01:57,104::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:01:57,108::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 16:01:58,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:01:58,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000642066 s, 6.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:01:59,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:01:59,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000945388 s, 4.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:02:07,123::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:02:07,124::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:02:07,563::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:02:07,569::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:02:07,570::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:02:08,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:02:08,494::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00321749 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:02:09,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:02:09,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00724066 s, 566 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:02:18,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:02:18,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181115 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:02:19,781::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:02:19,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832431 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:02:28,606::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:02:28,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014272 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:02:29,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:02:29,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876648 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:02:38,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:02:38,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807494 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:02:39,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:02:39,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737304 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:02:48,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:02:48,809::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146443 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:02:49,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:02:49,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149987 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:02:58,852::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 16:02:58,854::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 16:02:58,854::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:02:58,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00137061 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:02:58,887::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00137061 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 16:02:58,888::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 16:02:58,888::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 16:02:58,890::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 16:02:58,891::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:02:58,894::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:02:58,894::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:02:58,894::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:02:58,895::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:02:58,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:02:59,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:02:59,317::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:02:59,318::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 16:02:59,321::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 16:02:59,323::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 16:02:59,323::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 16:02:59,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:02:59,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:02:59,685::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:02:59,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00989818 s, 414 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:02:59,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:02:59,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0207961 s, 197 kB/s\n'; <rc> = 0
>Thread-14353::WARNING::2013-03-17 16:03:07,600::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:03:07,604::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 16:03:09,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:03:09,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792284 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:03:10,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:03:10,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879825 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:03:17,617::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:03:17,618::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:03:18,059::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:03:18,066::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:03:18,068::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:03:19,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:03:19,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0069885 s, 586 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:03:20,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:03:20,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150206 s, 273 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:03:29,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:03:29,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787459 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:03:30,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:03:30,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880729 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:03:39,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:03:40,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795165 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:03:40,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:03:40,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838651 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:03:50,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:03:50,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136778 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:03:50,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:03:50,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810173 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:04:00,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:04:00,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151487 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:04:00,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:04:00,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103088 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:04:10,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:04:10,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147265 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:04:10,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:04:10,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000991277 s, 4.1 MB/s\n'; <rc> = 0
>Thread-14354::WARNING::2013-03-17 16:04:18,112::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:04:18,115::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:04:20,352::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:04:20,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:04:20,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120565 s, 340 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:04:20,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151406 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:04:28,117::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:04:28,118::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:04:28,623::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:04:28,630::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:04:28,632::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:04:30,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:04:30,471::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:04:30,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172135 s, 238 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:04:30,529::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127235 s, 322 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:04:40,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:04:40,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757617 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:04:40,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:04:40,602::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00196979 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:04:50,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:04:50,561::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936172 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:04:50,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:04:50,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001529 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:05:00,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:05:00,611::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144666 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:05:00,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:05:00,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0590743 s, 69.3 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:05:10,617::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:05:10,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000719747 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:05:10,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:05:10,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00250689 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:05:20,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:05:20,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0213526 s, 192 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:05:20,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:05:20,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159056 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14355::WARNING::2013-03-17 16:05:28,670::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:05:28,674::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:05:30,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:05:30,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756254 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:05:31,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:05:31,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870222 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:05:38,677::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:05:38,679::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:05:39,121::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:05:39,128::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:05:39,129::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:05:40,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:05:40,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158233 s, 259 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:05:41,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:05:41,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115863 s, 354 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:05:50,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:05:50,837::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107343 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:05:51,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:05:51,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864339 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:06:00,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:06:00,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150622 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:06:01,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:06:01,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932353 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:06:10,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:06:10,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851371 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:06:11,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:06:11,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000903405 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:06:20,922::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 16:06:20,923::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 16:06:20,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:06:20,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000762905 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:06:20,956::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000762905 s, 2.7 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 16:06:20,956::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 16:06:20,956::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 16:06:20,958::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 16:06:20,959::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:06:20,960::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:06:20,960::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:06:20,960::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:06:20,961::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:06:20,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:06:21,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:06:21,397::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:06:21,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:06:21,402::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 16:06:21,450::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 16:06:21,455::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 16:06:21,459::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 16:06:21,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191974 s, 213 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:06:21,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:06:21,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:06:21,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:06:21,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108493 s, 378 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:06:31,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:06:31,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811836 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:06:31,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:06:31,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142877 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14356::WARNING::2013-03-17 16:06:39,156::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:06:39,160::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 16:06:41,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:06:41,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855268 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:06:42,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:06:42,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798234 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:06:49,161::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 16:06:49,163::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:06:49,164::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:06:49,631::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:06:49,638::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:06:49,639::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:06:51,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:06:51,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817149 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:06:52,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:06:52,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118305 s, 346 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:07:01,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:07:01,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820798 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:07:02,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:07:02,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00278535 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:07:11,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:07:11,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158994 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:07:12,128::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:07:12,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00213182 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:07:21,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:07:21,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135601 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:07:22,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:07:22,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879896 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:07:32,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:07:32,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000984435 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:07:32,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:07:32,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822543 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:07:42,070::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:07:42,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018628 s, 220 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:07:42,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:07:42,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00285447 s, 1.4 MB/s\n'; <rc> = 0
>Thread-14357::WARNING::2013-03-17 16:07:49,671::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:07:49,676::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 16:07:52,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:07:52,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862601 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:07:52,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:07:52,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085568 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:07:59,679::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:07:59,680::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:08:00,132::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:08:00,138::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:08:00,140::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:08:02,224::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 16:08:02,228::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 16:08:02,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:08:02,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000806441 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:08:02,256::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000806441 s, 2.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 16:08:02,257::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 16:08:02,257::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 16:08:02,259::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 16:08:02,261::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:08:02,262::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:08:02,262::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:08:02,263::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:08:02,263::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:08:02,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:08:02,350::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:08:02,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164523 s, 249 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:08:02,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:08:02,698::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:08:02,698::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 16:08:02,701::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 16:08:02,702::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 16:08:02,702::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 16:08:02,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:08:03,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:08:03,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:08:03,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00691696 s, 592 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:08:12,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:08:12,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144605 s, 283 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:08:13,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:08:13,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145809 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:08:22,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:08:22,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802624 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:08:23,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:08:23,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0007983 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:08:32,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:08:32,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746653 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:08:33,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:08:33,350::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000927517 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:08:42,537::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:08:42,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000750396 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:08:43,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:08:43,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148266 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:08:52,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:08:52,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802802 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:08:53,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:08:53,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203613 s, 201 kB/s\n'; <rc> = 0
>Thread-14358::WARNING::2013-03-17 16:09:00,166::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:09:00,170::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:09:02,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:09:02,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084074 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:09:03,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:09:03,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923792 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:09:10,172::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:09:10,173::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:09:10,627::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:09:10,634::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:09:10,635::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:09:12,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:09:12,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0195532 s, 209 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:09:13,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:09:13,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894614 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:09:22,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:09:22,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00740336 s, 553 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:09:23,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:09:23,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0210752 s, 194 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:09:32,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:09:32,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129807 s, 316 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:09:33,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:09:33,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142908 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:09:42,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:09:42,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149558 s, 274 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:09:43,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:09:43,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809707 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:09:52,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:09:52,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135611 s, 302 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:09:53,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:09:53,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139895 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:10:02,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:10:02,958::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759339 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:10:04,033::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:10:04,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019641 s, 209 kB/s\n'; <rc> = 0
>Thread-14359::WARNING::2013-03-17 16:10:10,674::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:10:10,679::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:10:12,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:10:12,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852164 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:10:14,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:10:14,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824848 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:10:20,682::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:10:20,683::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:10:21,137::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:10:21,144::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:10:21,146::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:10:23,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:10:23,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00365871 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:10:24,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:10:24,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131804 s, 311 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:10:33,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:10:33,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00096197 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:10:34,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:10:34,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083473 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:10:43,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:10:43,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840713 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:10:44,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:10:44,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.032685 s, 125 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:10:53,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:10:53,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836815 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:10:54,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:10:54,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000739037 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:11:03,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:11:03,188::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141848 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:11:04,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:11:04,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139404 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:11:13,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:11:13,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001584 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:11:14,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:11:14,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015764 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14360::WARNING::2013-03-17 16:11:21,175::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:11:21,181::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:11:23,245::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 16:11:23,246::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 16:11:23,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:11:23,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00104065 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:11:23,277::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00104065 s, 2.0 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 16:11:23,277::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 16:11:23,277::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 16:11:23,279::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 16:11:23,280::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:11:23,281::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:11:23,281::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:11:23,282::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:11:23,282::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:11:23,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:11:23,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:11:23,722::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:11:23,723::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 16:11:23,725::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 16:11:23,726::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 16:11:23,727::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 16:11:23,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:11:24,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:11:24,108::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:11:24,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175678 s, 233 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:11:24,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:11:24,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122521 s, 334 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:11:31,189::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:11:31,190::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:11:31,654::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:11:31,660::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:11:31,662::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:11:34,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:11:34,195::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00643827 s, 636 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:11:34,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:11:34,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158798 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:11:44,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:11:44,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158457 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:11:44,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:11:44,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001673 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:11:54,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:11:54,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159798 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:11:55,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:11:55,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923901 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:12:04,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:12:04,333::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116779 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:12:05,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:12:05,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138886 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:12:14,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:12:14,372::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150026 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:12:15,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:12:15,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758296 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:12:24,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:12:24,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105561 s, 388 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:12:25,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:12:25,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146292 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14361::WARNING::2013-03-17 16:12:31,691::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:12:31,697::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:12:34,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:12:34,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841951 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:12:35,324::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:12:35,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101464 s, 4.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:12:41,699::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 16:12:41,700::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:12:41,701::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:12:42,127::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:12:42,134::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:12:42,136::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:12:44,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:12:44,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121324 s, 338 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:12:45,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:12:45,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802841 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:12:54,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:12:54,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145007 s, 282 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:12:55,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:12:55,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855826 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:13:04,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:13:04,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151554 s, 270 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:13:05,547::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 16:13:05,549::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 16:13:05,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:13:05,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00085106 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:13:05,577::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00085106 s, 2.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 16:13:05,577::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 16:13:05,578::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 16:13:05,579::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 16:13:05,580::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:13:05,581::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:13:05,582::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:13:05,582::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:13:05,582::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:13:05,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:13:06,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:13:06,064::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:13:06,065::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 16:13:06,068::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 16:13:06,069::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 16:13:06,069::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 16:13:06,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:13:06,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:13:06,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:13:06,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00709938 s, 577 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:13:14,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:13:14,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185842 s, 220 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:13:16,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:13:16,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185962 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:13:24,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:13:24,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863432 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:13:26,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:13:26,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814492 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:13:34,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:13:34,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169465 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:13:36,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:13:36,692::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136824 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14362::WARNING::2013-03-17 16:13:42,162::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:13:42,166::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:13:44,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:13:44,786::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873687 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:13:46,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:13:46,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0226953 s, 180 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:13:52,167::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:13:52,169::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:13:52,647::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:13:52,654::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:13:52,655::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:13:54,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:13:54,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0084312 s, 486 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:13:56,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:13:56,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155544 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:14:04,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:14:04,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824803 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:14:06,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:14:06,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00522497 s, 784 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:14:14,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:14:14,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000948518 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:14:17,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:14:17,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000989277 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:14:24,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:14:24,958::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159113 s, 257 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:14:27,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:14:27,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197306 s, 208 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:14:34,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:14:35,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00289927 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:14:37,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:14:37,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013453 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:14:45,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:14:45,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00270569 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:14:47,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:14:47,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140861 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14363::WARNING::2013-03-17 16:14:52,686::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:14:52,695::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:14:55,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:14:55,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104917 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:14:57,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:14:57,382::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016293 s, 2.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:15:02,698::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:15:02,699::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:15:03,155::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:15:03,161::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:15:03,162::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:15:05,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:15:05,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017588 s, 233 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:15:07,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:15:07,450::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872775 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:15:15,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:15:15,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814901 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:15:17,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:15:17,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0232366 s, 176 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:15:25,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:15:25,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825913 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:15:27,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:15:27,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0224642 s, 182 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:15:35,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:15:35,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876084 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:15:37,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:15:37,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862074 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:15:45,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:15:45,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145665 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:15:47,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:15:47,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013908 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:15:55,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:15:55,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203568 s, 201 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:15:57,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:15:57,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741295 s, 5.5 MB/s\n'; <rc> = 0
>Thread-14364::WARNING::2013-03-17 16:16:03,193::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:16:03,197::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:16:05,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:16:05,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781413 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:16:07,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:16:07,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806253 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:16:13,199::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:16:13,201::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:16:13,676::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:16:13,683::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:16:13,685::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:16:15,492::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:16:15,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102598 s, 399 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:16:17,997::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:16:18,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00237097 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:16:25,559::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 16:16:25,560::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 16:16:25,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:16:25,591::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000846118 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:16:25,593::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000846118 s, 2.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 16:16:25,593::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 16:16:25,593::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 16:16:25,595::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 16:16:25,596::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:16:25,596::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:16:25,597::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:16:25,597::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:16:25,598::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:16:25,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:16:26,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:16:26,074::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:16:26,075::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 16:16:26,077::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 16:16:26,078::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 16:16:26,079::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 16:16:26,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:16:26,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:16:26,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:16:26,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00580096 s, 706 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:16:28,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:16:28,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158677 s, 258 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:16:36,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:16:36,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193089 s, 212 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:16:38,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:16:38,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907482 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:16:46,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:16:46,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151455 s, 270 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:16:48,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:16:48,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865981 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:16:56,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:16:56,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116596 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:16:58,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:16:58,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137908 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:17:06,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:17:06,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015016 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:17:08,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:17:08,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131224 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14365::WARNING::2013-03-17 16:17:13,712::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:17:13,717::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:17:16,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:17:16,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812459 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:17:18,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:17:18,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0216057 s, 190 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:17:23,720::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:17:23,722::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:17:24,163::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:17:24,172::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:17:24,174::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:17:26,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:17:26,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00703775 s, 582 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:17:28,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:17:28,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146497 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:17:36,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:17:36,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806987 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:17:38,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:17:38,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00309792 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:17:46,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:17:46,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150137 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:17:48,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:17:48,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129782 s, 316 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:17:56,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:17:56,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813881 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:17:58,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:17:58,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781999 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:18:06,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:18:06,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00187159 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:18:08,879::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 16:18:08,881::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 16:18:08,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:18:08,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.001036 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:18:08,919::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.001036 s, 2.0 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 16:18:08,919::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 16:18:08,920::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 16:18:08,921::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 16:18:08,922::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:18:08,923::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:18:08,924::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:18:08,924::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:18:08,924::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:18:08,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:18:09,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:18:09,425::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:18:09,426::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 16:18:09,428::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 16:18:09,429::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 16:18:09,430::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 16:18:09,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:18:09,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:18:09,778::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:18:09,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014733 s, 278 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:18:16,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:18:17,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149572 s, 274 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:18:19,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:18:19,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00689735 s, 594 kB/s\n'; <rc> = 0
>Thread-14366::WARNING::2013-03-17 16:18:24,204::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:18:24,209::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:18:27,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:18:27,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153841 s, 266 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:18:29,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:18:29,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795444 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:18:34,211::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 16:18:34,212::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:18:34,213::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:18:34,750::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:18:34,757::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:18:34,758::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:18:37,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:18:37,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177267 s, 231 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:18:40,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:18:40,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833552 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:18:47,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:18:47,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092293 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:18:50,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:18:50,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141456 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:18:57,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:18:57,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158836 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:19:00,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:19:00,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168746 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:19:07,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:19:07,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785711 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:19:10,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:19:10,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000889534 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:19:17,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:19:17,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016088 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:19:20,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:19:20,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148623 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:19:27,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:19:27,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00189935 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:19:30,396::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:19:30,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152213 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14367::WARNING::2013-03-17 16:19:34,789::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:19:34,794::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:19:37,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:19:37,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835693 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:19:40,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:19:40,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0223801 s, 183 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:19:44,797::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:19:44,798::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:19:45,225::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:19:45,232::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:19:45,234::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:19:47,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:19:47,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130498 s, 314 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:19:50,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:19:50,627::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164934 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:19:57,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:19:57,521::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158121 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:20:00,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:20:00,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00277078 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:20:07,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:20:07,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788332 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:20:10,738::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:20:10,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000960861 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:20:17,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:20:17,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791711 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:20:20,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:20:20,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111352 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:20:27,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:20:27,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121996 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:20:30,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:20:30,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147241 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:20:37,647::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:20:37,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010785 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:20:40,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:20:41,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854101 s, 4.8 MB/s\n'; <rc> = 0
>Thread-14368::WARNING::2013-03-17 16:20:45,273::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:20:45,279::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:20:47,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:20:47,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154903 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:20:51,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:20:51,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131843 s, 311 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:20:55,281::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:20:55,282::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:20:55,786::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:20:55,793::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:20:55,794::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:20:57,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:20:57,777::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00712228 s, 575 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:21:01,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:21:01,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854009 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:21:07,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:21:07,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000993263 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:21:11,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:21:11,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158703 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:21:17,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:21:17,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147677 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:21:21,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:21:21,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00614839 s, 666 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:21:27,874::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 16:21:27,875::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 16:21:27,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:21:27,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0013739 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:21:27,907::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0013739 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 16:21:27,907::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 16:21:27,908::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 16:21:27,909::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 16:21:27,910::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:21:27,911::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:21:27,911::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:21:27,912::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:21:27,912::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:21:27,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:21:28,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:21:28,351::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:21:28,352::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 16:21:28,355::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 16:21:28,356::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 16:21:28,356::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 16:21:28,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:21:28,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:21:28,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:21:28,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168626 s, 243 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:21:31,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:21:31,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769933 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:21:38,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:21:38,820::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819861 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:21:41,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:21:41,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816805 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:21:48,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:21:48,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131663 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:21:51,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:21:51,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001116 s, 3.7 MB/s\n'; <rc> = 0
>Thread-14369::WARNING::2013-03-17 16:21:55,823::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:21:55,829::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:21:58,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:21:58,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00256429 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:22:01,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:22:01,658::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0224388 s, 183 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:22:05,839::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:22:05,840::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:22:06,256::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:22:06,263::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:22:06,264::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:22:08,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:22:08,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106191 s, 386 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:22:11,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:22:11,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140392 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:22:18,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:22:19,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148554 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:22:21,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:22:21,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145119 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:22:29,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:22:29,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136411 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:22:31,875::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:22:31,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758131 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:22:39,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:22:39,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146204 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:22:41,944::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:22:41,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191003 s, 214 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:22:49,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:22:49,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001796 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:22:52,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:22:52,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0306107 s, 134 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:22:59,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:22:59,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140532 s, 291 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:23:02,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:23:02,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00191824 s, 2.1 MB/s\n'; <rc> = 0
>Thread-14370::WARNING::2013-03-17 16:23:06,293::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:23:06,297::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:23:09,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:23:09,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896761 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:23:12,224::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 16:23:12,225::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 16:23:12,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:23:12,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000795169 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:23:12,253::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000795169 s, 2.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 16:23:12,253::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 16:23:12,254::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 16:23:12,255::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 16:23:12,256::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:23:12,257::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:23:12,258::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:23:12,258::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:23:12,258::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:23:12,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:23:12,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:23:12,689::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:23:12,690::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 16:23:12,692::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 16:23:12,693::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 16:23:12,694::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 16:23:12,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:23:13,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:23:13,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:23:13,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00449432 s, 911 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:23:16,299::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:23:16,301::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:23:16,756::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:23:16,762::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:23:16,764::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:23:19,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:23:19,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00922233 s, 444 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:23:23,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:23:23,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102826 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:23:29,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:23:29,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874362 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:23:33,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:23:33,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807595 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:23:39,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:23:39,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776576 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:23:43,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:23:43,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138421 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:23:49,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:23:49,398::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872236 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:23:53,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:23:53,364::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866698 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:23:59,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:23:59,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000908984 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:24:03,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:24:03,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796207 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:24:09,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:24:09,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121931 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:24:13,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:24:13,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00484575 s, 845 kB/s\n'; <rc> = 0
>Thread-14371::WARNING::2013-03-17 16:24:16,794::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:24:16,799::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:24:19,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:24:19,529::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181138 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:24:23,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:24:23,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757543 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:24:26,808::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 16:24:26,809::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:24:26,810::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:24:27,247::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:24:27,253::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:24:27,255::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:24:29,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:24:29,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018135 s, 226 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:24:33,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:24:33,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000941155 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:24:39,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:24:39,627::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101074 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:24:43,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:24:43,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014139 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:24:49,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:24:49,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147846 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:24:53,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:24:53,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142688 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:24:59,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:24:59,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138532 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:25:03,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:25:03,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014505 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:25:09,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:25:09,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160659 s, 255 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:25:13,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:25:13,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000720888 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:25:19,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:25:19,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017493 s, 234 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:25:24,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:25:24,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113708 s, 360 kB/s\n'; <rc> = 0
>Thread-14372::WARNING::2013-03-17 16:25:27,281::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:25:27,285::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:25:29,875::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:25:29,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000945598 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:25:34,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:25:34,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0209638 s, 195 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:25:37,287::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:25:37,288::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:25:37,722::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:25:37,728::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:25:37,730::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:25:39,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:25:39,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00795128 s, 515 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:25:44,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:25:44,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897873 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:25:49,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:25:50,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197645 s, 207 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:25:54,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:25:54,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082189 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:26:00,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:26:00,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149031 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:26:04,416::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:26:04,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852096 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:26:10,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:26:10,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743526 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:26:14,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:26:14,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130036 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:26:20,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:26:20,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00713516 s, 574 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:26:24,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:26:24,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089492 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:26:30,152::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 16:26:30,154::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 16:26:30,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:26:30,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000858736 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:26:30,192::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000858736 s, 2.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 16:26:30,192::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 16:26:30,192::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 16:26:30,194::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 16:26:30,195::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:26:30,196::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:26:30,196::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:26:30,196::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:26:30,197::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:26:30,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:26:30,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:26:30,660::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:26:30,661::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 16:26:30,663::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 16:26:30,664::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 16:26:30,665::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 16:26:30,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:26:31,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:26:31,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:26:31,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0195302 s, 210 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:26:34,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:26:34,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199835 s, 205 kB/s\n'; <rc> = 0
>Thread-14373::WARNING::2013-03-17 16:26:37,759::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:26:37,765::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:26:41,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:26:41,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746954 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:26:44,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:26:44,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189492 s, 216 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:26:47,768::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:26:47,769::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:26:48,235::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:26:48,242::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:26:48,243::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:26:51,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:26:51,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00564574 s, 726 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:26:54,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:26:54,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083722 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:27:01,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:27:01,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0026776 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:27:04,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:27:04,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173133 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:27:11,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:27:11,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156518 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:27:14,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:27:15,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132321 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:27:21,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:27:21,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152211 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:27:25,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:27:25,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867241 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:27:31,365::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:27:31,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139211 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:27:35,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:27:35,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149609 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:27:41,412::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:27:41,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000989652 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:27:45,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:27:45,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145335 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14374::WARNING::2013-03-17 16:27:48,269::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:27:48,272::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:27:51,465::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:27:51,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823203 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:27:55,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:27:55,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906883 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:27:58,275::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:27:58,276::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:27:58,732::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:27:58,738::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:27:58,740::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:28:01,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:28:01,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138034 s, 297 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:28:05,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:28:05,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109634 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:28:11,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:28:11,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849751 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:28:15,439::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 16:28:15,440::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 16:28:15,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:28:15,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0231805 s, 88.3 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:28:15,503::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0231805 s, 88.3 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 16:28:15,504::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 16:28:15,505::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 16:28:15,506::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 16:28:15,507::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:28:15,508::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:28:15,508::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:28:15,509::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:28:15,509::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:28:15,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:28:15,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:28:15,925::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:28:15,926::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 16:28:15,929::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 16:28:15,930::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 16:28:15,931::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 16:28:15,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:28:16,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:28:16,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:28:16,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00771702 s, 531 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:28:21,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:28:21,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000734519 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:28:26,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:28:26,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000921708 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:28:31,625::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:28:31,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779127 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:28:36,481::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:28:36,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840871 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:28:41,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:28:41,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148094 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:28:46,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:28:46,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151419 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:28:51,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:28:51,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165189 s, 248 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:28:56,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:28:56,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741466 s, 5.5 MB/s\n'; <rc> = 0
>Thread-14375::WARNING::2013-03-17 16:28:58,785::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:28:58,787::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:29:01,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:29:01,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166762 s, 246 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:29:06,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:29:06,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756137 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:29:08,790::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:29:08,791::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:29:09,270::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:29:09,276::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:29:09,278::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:29:11,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:29:11,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00893254 s, 459 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:29:16,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:29:16,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192611 s, 213 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:29:21,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:29:21,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013943 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:29:26,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:29:26,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139546 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:29:31,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:29:31,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147308 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:29:36,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:29:37,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858687 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:29:41,959::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:29:41,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887724 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:29:47,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:29:47,081::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838089 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:29:51,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:29:52,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168692 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:29:57,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:29:57,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775737 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:30:02,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:30:02,081::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00096358 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:30:07,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:30:07,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00303822 s, 1.3 MB/s\n'; <rc> = 0
>Thread-14376::WARNING::2013-03-17 16:30:09,304::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:30:09,307::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:30:12,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:30:12,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785212 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:30:17,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:30:17,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882932 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:30:19,310::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 16:30:19,311::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:30:19,312::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:30:19,809::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:30:19,816::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:30:19,817::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:30:22,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:30:22,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133671 s, 306 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:30:27,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:30:27,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785864 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:30:32,178::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:30:32,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835322 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:30:37,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:30:37,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140232 s, 292 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:30:42,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:30:42,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832382 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:30:47,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:30:47,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155335 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:30:52,254::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:30:52,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877138 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:30:57,592::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:30:57,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000679953 s, 6.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:31:02,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:31:02,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155027 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:31:07,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:31:07,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143022 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:31:12,341::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:31:12,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017564 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:31:17,738::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:31:17,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103114 s, 4.0 MB/s\n'; <rc> = 0
>Thread-14377::WARNING::2013-03-17 16:31:19,850::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:31:19,854::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:31:22,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:31:22,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775505 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:31:27,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:31:27,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164366 s, 2.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:31:29,856::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:31:29,857::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:31:30,343::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:31:30,350::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:31:30,351::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:31:32,422::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 16:31:32,425::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 16:31:32,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:31:32,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0144554 s, 142 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:31:32,473::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0144554 s, 142 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 16:31:32,474::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 16:31:32,474::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 16:31:32,476::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 16:31:32,477::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:31:32,478::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:31:32,479::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:31:32,479::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:31:32,480::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:31:32,481::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:31:32,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:31:32,866::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:31:32,867::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 16:31:32,869::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 16:31:32,871::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 16:31:32,871::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 16:31:32,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:31:33,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:31:33,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:31:33,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176307 s, 232 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:31:37,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:31:37,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849954 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:31:43,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:31:43,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885493 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:31:47,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:31:48,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136175 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:31:53,312::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:31:53,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109358 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:31:58,048::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:31:58,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144665 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:32:03,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:32:03,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142404 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:32:08,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:32:08,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846661 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:32:13,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:32:13,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000688228 s, 6.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:32:18,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:32:18,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150305 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:32:23,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:32:23,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000964411 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:32:28,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:32:28,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160722 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14378::WARNING::2013-03-17 16:32:30,379::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:32:30,382::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:32:33,481::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:32:33,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806621 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:32:38,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:32:38,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818709 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:32:40,384::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:32:40,385::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:32:40,822::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:32:40,829::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:32:40,830::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:32:43,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:32:43,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108833 s, 376 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:32:48,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:32:48,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010278 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:32:53,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:32:53,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814694 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:32:58,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:32:58,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776605 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:33:03,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:33:03,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00786566 s, 521 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:33:08,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:33:08,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762694 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:33:13,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:33:13,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829168 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:33:18,637::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 16:33:18,639::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 16:33:18,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:33:18,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00132858 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:33:18,684::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00132858 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 16:33:18,685::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 16:33:18,685::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 16:33:18,687::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 16:33:18,689::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:33:18,690::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:33:18,691::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:33:18,691::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:33:18,691::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:33:18,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:33:19,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:33:19,112::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:33:19,113::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 16:33:19,116::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 16:33:19,117::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 16:33:19,117::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 16:33:19,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:33:19,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:33:19,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:33:19,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00794957 s, 515 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:33:23,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:33:23,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105918 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:33:29,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:33:29,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0208967 s, 196 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:33:33,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:33:33,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181281 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:33:39,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:33:39,709::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826308 s, 5.0 MB/s\n'; <rc> = 0
>Thread-14379::WARNING::2013-03-17 16:33:40,875::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:33:40,877::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:33:43,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:33:43,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00445594 s, 919 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:33:50,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:33:50,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832152 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:33:50,941::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:33:50,943::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:33:51,353::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:33:51,359::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:33:51,361::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:33:53,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:33:53,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00561325 s, 730 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:34:00,230::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:34:00,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155158 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:34:03,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:34:03,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189099 s, 217 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:34:10,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:34:10,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762155 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:34:13,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:34:13,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795672 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:34:20,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:34:20,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864647 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:34:23,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:34:24,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118635 s, 345 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:34:30,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:34:30,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00183365 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:34:34,034::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:34:34,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156435 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:34:40,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:34:40,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806756 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:34:44,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:34:44,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.012505 s, 328 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:34:50,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:34:50,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0085828 s, 477 kB/s\n'; <rc> = 0
>Thread-14380::WARNING::2013-03-17 16:34:51,387::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:34:51,391::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:34:54,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:34:54,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108108 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:35:01,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:35:01,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.02705 s, 151 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:35:01,394::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:35:01,395::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:35:01,821::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:35:01,828::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:35:01,830::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:35:04,221::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:35:04,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014603 s, 280 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:35:11,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:35:11,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875964 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:35:14,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:35:14,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159435 s, 257 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:35:21,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:35:21,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100924 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:35:24,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:35:24,392::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014105 s, 290 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:35:31,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:35:31,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206721 s, 198 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:35:34,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:35:34,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148847 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:35:41,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:35:41,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177601 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:35:44,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:35:44,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110228 s, 372 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:35:51,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:35:51,607::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881877 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:35:54,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:35:54,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797448 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:36:01,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:36:01,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122611 s, 3.3 MB/s\n'; <rc> = 0
>Thread-14381::WARNING::2013-03-17 16:36:01,857::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:36:01,870::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:36:04,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:36:04,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864758 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:36:11,892::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 16:36:11,893::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:36:11,894::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:36:11,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:36:12,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0235646 s, 174 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:36:12,413::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:36:12,419::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:36:12,420::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:36:14,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:36:14,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100472 s, 408 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:36:22,070::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:36:22,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110712 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:36:24,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:36:24,649::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850951 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:36:32,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:36:32,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0314475 s, 130 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:36:34,654::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 16:36:34,657::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 16:36:34,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:36:34,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00109117 s, 1.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:36:34,689::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00109117 s, 1.9 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 16:36:34,689::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 16:36:34,689::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 16:36:34,692::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 16:36:34,693::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:36:34,694::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:36:34,694::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:36:34,694::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:36:34,695::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:36:34,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:36:35,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:36:35,099::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:36:35,100::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 16:36:35,102::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 16:36:35,104::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 16:36:35,104::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 16:36:35,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:36:35,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:36:35,455::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:36:35,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151337 s, 271 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:36:42,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:36:42,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0216566 s, 189 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:36:45,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:36:45,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135771 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:36:52,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:36:52,358::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787692 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:36:55,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:36:55,598::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145814 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:37:02,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:37:02,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813048 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:37:05,603::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:37:05,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00198534 s, 2.1 MB/s\n'; <rc> = 0
>Thread-14382::WARNING::2013-03-17 16:37:12,464::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:37:12,471::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 16:37:12,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:37:12,517::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780304 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:37:15,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:37:15,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802086 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:37:22,482::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:37:22,484::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:37:22,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:37:22,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120028 s, 341 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:37:22,964::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:37:22,970::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:37:22,971::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:37:25,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:37:25,738::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106106 s, 386 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:37:32,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:37:32,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111425 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:37:35,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:37:35,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000958595 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:37:42,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:37:42,791::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901315 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:37:45,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:37:45,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754673 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:37:52,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:37:52,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904283 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:37:55,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:37:55,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080623 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:38:02,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:38:02,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137424 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:38:05,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:38:05,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155623 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:38:12,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:38:13,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172362 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:38:15,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:38:15,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173273 s, 236 kB/s\n'; <rc> = 0
>Thread-14383::WARNING::2013-03-17 16:38:22,997::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:38:23,004::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 16:38:23,073::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 16:38:23,074::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 16:38:23,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:38:23,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000836515 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:38:23,101::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000836515 s, 2.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 16:38:23,101::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 16:38:23,102::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 16:38:23,103::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 16:38:23,104::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:38:23,105::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:38:23,106::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:38:23,106::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:38:23,106::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:38:23,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:38:23,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:38:23,534::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:38:23,536::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 16:38:23,538::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 16:38:23,539::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 16:38:23,540::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 16:38:23,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:38:23,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:38:23,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:38:23,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00316028 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:38:25,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:38:25,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164795 s, 249 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:38:33,007::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:38:33,008::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:38:33,414::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:38:33,422::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:38:33,423::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:38:33,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:38:34,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0217407 s, 188 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:38:36,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:38:36,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00840123 s, 488 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:38:44,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:38:44,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111959 s, 366 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:38:46,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:38:46,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146248 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:38:54,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:38:54,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160565 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:38:56,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:38:56,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163239 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:39:04,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:39:04,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129218 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:39:06,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:39:06,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769725 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:39:14,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:39:14,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818763 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:39:16,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:39:16,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162669 s, 252 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:39:24,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:39:24,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113768 s, 360 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:39:26,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:39:26,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786937 s, 5.2 MB/s\n'; <rc> = 0
>Thread-14384::WARNING::2013-03-17 16:39:33,452::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:39:33,455::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 16:39:34,465::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:39:34,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102128 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:39:36,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:39:36,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170867 s, 240 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:39:43,457::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:39:43,458::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:39:43,882::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:39:43,889::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:39:43,890::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:39:44,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:39:44,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136071 s, 301 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:39:46,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:39:46,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182728 s, 224 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:39:54,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:39:54,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001441 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:39:56,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:39:56,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867642 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:40:04,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:40:04,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823123 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:40:06,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:40:06,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015967 s, 257 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:40:14,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:40:14,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106758 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:40:16,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:40:16,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134158 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:40:24,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:40:24,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179914 s, 228 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:40:26,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:40:26,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748224 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:40:34,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:40:34,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162596 s, 252 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:40:36,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:40:36,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134651 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14385::WARNING::2013-03-17 16:40:43,918::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:40:43,921::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 16:40:45,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:40:45,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772221 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:40:46,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:40:46,657::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000731472 s, 5.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:40:53,934::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:40:53,935::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:40:54,355::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:40:54,362::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:40:54,363::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:40:55,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:40:55,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013987 s, 293 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:40:56,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:40:56,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018269 s, 224 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:41:05,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:41:05,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165938 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:41:06,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:41:06,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786305 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:41:15,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:41:15,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848083 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:41:16,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:41:16,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109205 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:41:25,345::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:41:25,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103967 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:41:26,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:41:26,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00628253 s, 652 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:41:35,416::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:41:35,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158486 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:41:36,851::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 16:41:36,852::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 16:41:36,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:41:36,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000894154 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:41:36,885::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000894154 s, 2.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 16:41:36,885::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 16:41:36,886::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 16:41:36,888::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 16:41:36,889::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:41:36,890::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:41:36,890::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:41:36,893::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:41:36,894::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:41:36,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:41:37,311::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:41:37,316::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:41:37,317::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 16:41:37,319::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 16:41:37,321::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 16:41:37,321::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 16:41:37,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:41:37,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:41:37,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:41:37,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0081639 s, 502 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:41:45,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:41:45,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793811 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:41:47,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:41:47,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873515 s, 4.7 MB/s\n'; <rc> = 0
>Thread-14386::WARNING::2013-03-17 16:41:54,389::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:41:54,393::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 16:41:55,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:41:55,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757008 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:41:57,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:41:57,818::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786918 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:42:04,394::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 16:42:04,396::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:42:04,397::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:42:04,919::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:42:04,926::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:42:04,927::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:42:05,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:42:05,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148088 s, 277 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:42:07,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:42:07,866::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129423 s, 316 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:42:15,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:42:15,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00856742 s, 478 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:42:17,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:42:17,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000729909 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:42:25,857::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:42:25,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802706 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:42:27,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:42:27,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00183855 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:42:35,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:42:35,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118881 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:42:37,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:42:38,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892272 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:42:46,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:42:46,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014874 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:42:48,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:42:48,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147164 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:42:56,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:42:56,175::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136506 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:42:58,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:42:58,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764082 s, 5.4 MB/s\n'; <rc> = 0
>Thread-14387::WARNING::2013-03-17 16:43:04,964::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:43:04,967::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 16:43:06,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:43:06,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777483 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:43:08,117::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:43:08,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000942344 s, 4.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:43:14,973::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:43:14,974::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:43:15,408::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:43:15,414::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:43:15,416::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:43:16,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:43:16,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147665 s, 277 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:43:18,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:43:18,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127464 s, 321 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:43:26,372::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 16:43:26,374::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 16:43:26,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:43:26,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00105598 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:43:26,406::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00105598 s, 1.9 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 16:43:26,407::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 16:43:26,407::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 16:43:26,409::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 16:43:26,410::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:43:26,410::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:43:26,411::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:43:26,411::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:43:26,412::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:43:26,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:43:26,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:43:26,832::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:43:26,833::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 16:43:26,837::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 16:43:26,838::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 16:43:26,839::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 16:43:26,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:43:27,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:43:27,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:43:27,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183073 s, 224 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:43:28,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:43:28,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113895 s, 360 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:43:37,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:43:37,358::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874336 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:43:38,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:43:38,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129428 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:43:47,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:43:47,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000719427 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:43:48,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:43:48,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000977071 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:43:57,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:43:57,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144767 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:43:58,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:43:58,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116652 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:44:07,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:44:07,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799098 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:44:08,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:44:08,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137458 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14388::WARNING::2013-03-17 16:44:15,447::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:44:15,450::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 16:44:17,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:44:17,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755257 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:44:18,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:44:18,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764727 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:44:25,452::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:44:25,453::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:44:25,878::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:44:25,884::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:44:25,886::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:44:27,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:44:27,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111584 s, 367 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:44:28,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:44:28,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135439 s, 302 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:44:37,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:44:37,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811069 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:44:38,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:44:38,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155656 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:44:47,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:44:47,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925889 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:44:48,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:44:48,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146093 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:44:57,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:44:57,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0240948 s, 170 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:44:58,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:44:58,667::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152637 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:45:08,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:45:08,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077472 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:45:08,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:45:08,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00330763 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:45:18,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:45:18,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794437 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:45:18,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:45:18,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135536 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14389::WARNING::2013-03-17 16:45:25,922::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:45:25,925::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 16:45:28,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:45:28,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076414 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:45:28,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:45:28,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165323 s, 248 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:45:35,931::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:45:35,939::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:45:36,399::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:45:36,405::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:45:36,407::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:45:38,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:45:38,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204759 s, 200 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:45:38,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:45:38,866::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170655 s, 240 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:45:48,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:45:48,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0208003 s, 197 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:45:48,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:45:48,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155139 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:45:58,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:45:58,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0201553 s, 203 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:45:58,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:45:58,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138196 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:46:08,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:46:08,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185065 s, 221 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:46:08,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:46:09,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00099101 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:46:18,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:46:18,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150862 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:46:19,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:46:19,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112978 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:46:28,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:46:28,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858283 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:46:29,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:46:29,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138871 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14390::WARNING::2013-03-17 16:46:36,433::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:46:36,437::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 16:46:38,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:46:38,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859513 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:46:39,191::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 16:46:39,192::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 16:46:39,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:46:39,222::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0007951 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:46:39,224::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0007951 s, 2.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 16:46:39,224::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 16:46:39,224::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 16:46:39,226::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 16:46:39,227::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:46:39,228::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:46:39,228::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:46:39,229::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:46:39,229::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:46:39,230::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:46:39,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:46:39,664::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:46:39,665::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 16:46:39,667::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 16:46:39,669::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 16:46:39,669::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 16:46:39,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:46:40,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:46:40,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:46:40,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124397 s, 329 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:46:46,439::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:46:46,440::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:46:46,894::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:46:46,900::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:46:46,902::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:46:48,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:46:48,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144631 s, 283 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:46:50,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:46:50,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163553 s, 250 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:46:58,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:46:59,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000750729 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:47:00,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:47:00,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175712 s, 233 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:47:09,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:47:09,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0224193 s, 183 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:47:10,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:47:10,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018773 s, 218 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:47:19,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:47:19,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000701189 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:47:20,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:47:20,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783784 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:47:29,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:47:29,287::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0198655 s, 206 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:47:30,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:47:30,331::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181371 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:47:39,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:47:39,358::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749288 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:47:40,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:47:40,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015301 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14391::WARNING::2013-03-17 16:47:46,931::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:47:46,933::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 16:47:49,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:47:49,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076623 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:47:50,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:47:50,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837816 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:47:56,942::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 16:47:56,943::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:47:56,945::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:47:57,415::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:47:57,421::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:47:57,423::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:47:59,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:47:59,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140799 s, 291 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:48:00,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:48:00,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128349 s, 319 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:48:09,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:48:09,591::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151465 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:48:10,471::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:48:10,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101136 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:48:19,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:48:19,657::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740613 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:48:20,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:48:20,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759458 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:48:29,702::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 16:48:29,704::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 16:48:29,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:48:29,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00133887 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:48:29,741::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00133887 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 16:48:29,742::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 16:48:29,742::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 16:48:29,744::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 16:48:29,745::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:48:29,745::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:48:29,746::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:48:29,746::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:48:29,746::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:48:29,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:48:30,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:48:30,174::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:48:30,176::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 16:48:30,178::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 16:48:30,179::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 16:48:30,180::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 16:48:30,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:48:30,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:48:30,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:48:30,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:48:30,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152442 s, 269 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:48:30,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108562 s, 377 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:48:40,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:48:40,681::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142173 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:48:40,697::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:48:40,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00184753 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:48:50,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:48:50,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146314 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:48:50,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:48:50,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147533 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14392::WARNING::2013-03-17 16:48:57,453::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:48:57,459::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:49:00,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:49:00,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859834 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:49:00,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:49:00,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00069084 s, 5.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:49:07,463::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:49:07,464::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:49:07,925::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:49:07,931::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:49:07,933::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:49:10,792::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:49:10,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00827356 s, 495 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:49:10,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:49:10,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130319 s, 314 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:49:20,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:49:20,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748295 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:49:21,014::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:49:21,040::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000718671 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:49:30,875::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:49:30,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902909 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:49:31,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:49:31,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154189 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:49:40,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:49:40,944::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081235 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:49:41,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:49:41,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082438 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:49:50,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:49:50,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152318 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:49:51,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:49:51,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017058 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:50:01,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:50:01,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144336 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:50:01,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:50:01,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.012241 s, 335 kB/s\n'; <rc> = 0
>Thread-14393::WARNING::2013-03-17 16:50:07,961::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:50:07,967::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:50:11,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:50:11,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109803 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:50:11,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:50:11,636::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194604 s, 210 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:50:17,970::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:50:17,971::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:50:18,545::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:50:18,552::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:50:18,553::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:50:21,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:50:21,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0223809 s, 183 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:50:21,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:50:21,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185283 s, 221 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:50:31,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:50:31,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124935 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:50:31,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:50:31,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184483 s, 222 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:50:41,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:50:41,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:50:41,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:50:41,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017636 s, 232 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:50:51,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:50:51,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169258 s, 242 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:50:51,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:50:51,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841707 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:51:01,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:51:01,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0251277 s, 163 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:51:02,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:51:02,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146065 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:51:11,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:51:11,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137513 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:51:12,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:51:12,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153165 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14394::WARNING::2013-03-17 16:51:18,579::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:51:18,583::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:51:21,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:51:21,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886793 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:51:22,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:51:22,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906977 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:51:28,588::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:51:28,590::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:51:29,068::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:51:29,074::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:51:29,076::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:51:31,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:51:31,512::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155436 s, 264 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:51:32,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:51:32,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0347193 s, 118 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:51:41,520::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 16:51:41,522::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 16:51:41,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:51:41,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000911044 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:51:41,555::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000911044 s, 2.2 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 16:51:41,556::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 16:51:41,556::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 16:51:41,557::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 16:51:41,559::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:51:41,559::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:51:41,560::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:51:41,560::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:51:41,560::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:51:41,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:51:42,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:51:42,005::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:51:42,006::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 16:51:42,009::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 16:51:42,010::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 16:51:42,010::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 16:51:42,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:51:42,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:51:42,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:51:42,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:51:42,450::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122804 s, 334 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:51:42,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154205 s, 266 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:51:52,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:51:52,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:51:52,569::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114568 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:51:52,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860732 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:52:02,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:52:02,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:52:02,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749896 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:52:02,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000693911 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:52:12,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:52:12,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766568 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:52:12,732::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:52:12,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0235334 s, 174 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:52:22,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:52:22,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151239 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:52:22,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:52:22,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146058 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14395::WARNING::2013-03-17 16:52:29,116::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:52:29,120::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:52:32,781::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:52:32,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159358 s, 257 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:52:32,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:52:32,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824983 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:52:39,122::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:52:39,123::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:52:39,569::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:52:39,575::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:52:39,577::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:52:42,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:52:42,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146137 s, 280 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:52:43,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:52:43,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017374 s, 236 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:52:52,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:52:52,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791549 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:52:53,092::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:52:53,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0228339 s, 179 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:53:02,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:53:02,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203943 s, 201 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:53:03,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:53:03,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156278 s, 262 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:53:12,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:53:13,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154935 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:53:13,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:53:13,347::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169494 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:53:23,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:53:23,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754606 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:53:23,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:53:23,424::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849474 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:53:33,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:53:33,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131853 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:53:33,462::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 16:53:33,464::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 16:53:33,465::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:53:33,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00158107 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:53:33,504::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00158107 s, 1.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 16:53:33,504::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 16:53:33,505::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 16:53:33,507::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 16:53:33,508::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:53:33,508::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:53:33,509::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:53:33,509::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:53:33,510::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:53:33,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:53:33,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:53:33,922::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:53:33,925::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 16:53:33,928::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 16:53:33,929::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 16:53:33,930::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 16:53:33,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:53:34,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:53:34,292::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:53:34,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00499275 s, 820 kB/s\n'; <rc> = 0
>Thread-14396::WARNING::2013-03-17 16:53:39,607::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:53:39,609::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:53:43,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:53:43,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767993 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:53:44,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:53:44,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0306399 s, 134 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:53:49,611::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 16:53:49,612::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:53:49,613::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:53:50,057::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:53:50,064::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:53:50,065::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:53:53,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:53:53,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121283 s, 338 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:53:54,482::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:53:54,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161034 s, 254 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:54:03,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:54:03,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802093 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:54:04,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:54:04,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187094 s, 219 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:54:13,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:54:13,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827549 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:54:14,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:54:14,700::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136328 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:54:23,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:54:23,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000699746 s, 5.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:54:24,742::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:54:24,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193144 s, 212 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:54:33,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:54:33,347::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00217798 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:54:34,837::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:54:34,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148044 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:54:43,352::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:54:43,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806421 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:54:44,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:54:44,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177881 s, 2.3 MB/s\n'; <rc> = 0
>Thread-14397::WARNING::2013-03-17 16:54:50,127::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:54:50,142::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:54:53,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:54:53,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122929 s, 333 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:54:55,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:54:55,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749041 s, 5.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:55:00,145::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:55:00,146::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:55:00,639::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:55:00,645::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:55:00,647::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:55:03,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:55:03,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00947918 s, 432 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:55:05,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:55:05,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801408 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:55:13,492::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:55:13,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165004 s, 248 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:55:15,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:55:15,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877039 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:55:23,571::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:55:23,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00310029 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:55:25,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:55:25,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102952 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:55:33,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:55:33,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149367 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:55:35,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:55:35,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847625 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:55:43,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:55:43,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912045 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:55:45,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:55:45,392::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140011 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:55:53,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:55:53,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166232 s, 246 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:55:55,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:55:55,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183806 s, 223 kB/s\n'; <rc> = 0
>Thread-14398::WARNING::2013-03-17 16:56:00,675::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:56:00,678::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:56:03,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:56:03,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836947 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:56:05,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:56:05,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078648 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:56:10,681::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:56:10,682::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:56:11,186::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:56:11,193::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:56:11,195::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:56:13,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:56:13,843::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204397 s, 200 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:56:15,603::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:56:15,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.004725 s, 867 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:56:23,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:56:23,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860118 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:56:25,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:56:25,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765874 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:56:33,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:56:33,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147439 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:56:35,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:56:35,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000691983 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:56:43,928::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 16:56:43,929::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 16:56:43,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:56:43,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000732418 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:56:43,963::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000732418 s, 2.8 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 16:56:43,964::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 16:56:43,964::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 16:56:43,967::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 16:56:43,968::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:56:43,969::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:56:43,969::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:56:43,970::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:56:43,970::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 16:56:43,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:56:44,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:56:44,426::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:56:44,427::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 16:56:44,429::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 16:56:44,430::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 16:56:44,431::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 16:56:44,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:56:44,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:56:44,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:56:44,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188054 s, 218 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:56:45,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:56:45,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140986 s, 291 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:56:54,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:56:54,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140033 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:56:55,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:56:55,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150493 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:57:04,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:57:04,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836608 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:57:06,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:57:06,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00468876 s, 874 kB/s\n'; <rc> = 0
>Thread-14399::WARNING::2013-03-17 16:57:11,220::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:57:11,224::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:57:14,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:57:14,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167926 s, 244 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:57:16,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:57:16,130::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139446 s, 2.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:57:21,226::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:57:21,227::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:57:21,722::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:57:21,729::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:57:21,731::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:57:24,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:57:25,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103171 s, 397 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:57:26,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:57:26,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015475 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:57:35,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:57:35,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834459 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:57:36,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:57:36,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000950387 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:57:45,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:57:45,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859351 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:57:46,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:57:46,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105354 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:57:55,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:57:55,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811168 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:57:56,395::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:57:56,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108554 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:58:05,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:58:05,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00095618 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:58:06,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:58:06,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149216 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:58:15,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:58:15,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748903 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:58:16,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:58:16,591::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153583 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14400::WARNING::2013-03-17 16:58:21,797::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:58:21,800::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:58:25,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:58:25,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179839 s, 228 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:58:26,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:58:26,681::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000926994 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:58:31,803::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:58:31,804::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:58:32,261::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:58:32,268::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:58:32,269::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:58:35,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:58:35,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148933 s, 275 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:58:36,720::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 16:58:36,722::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 16:58:36,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:58:36,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000816587 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:58:36,748::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000816587 s, 2.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 16:58:36,748::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 16:58:36,748::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 16:58:36,750::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 16:58:36,751::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:58:36,752::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:58:36,752::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:58:36,752::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:58:36,753::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 16:58:36,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:58:37,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:58:37,167::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 16:58:37,168::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 16:58:37,170::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 16:58:37,171::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 16:58:37,172::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 16:58:37,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:58:37,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:58:37,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:58:37,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101477 s, 404 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:58:45,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:58:45,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880119 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:58:47,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:58:47,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014529 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:58:55,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:58:55,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899506 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:58:57,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:58:57,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861735 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:59:05,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:59:05,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103311 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:59:07,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:59:07,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145455 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:59:15,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:59:15,478::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831667 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:59:17,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:59:17,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839102 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:59:25,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:59:25,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101976 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:59:27,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:59:27,992::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0215833 s, 190 kB/s\n'; <rc> = 0
>Thread-14401::WARNING::2013-03-17 16:59:32,332::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 16:59:32,338::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 16:59:35,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:59:35,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172157 s, 238 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:59:38,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:59:38,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790188 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 16:59:42,341::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 16:59:42,342::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 16:59:42,343::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 16:59:42,967::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 16:59:42,973::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 16:59:42,974::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 16:59:45,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:59:45,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0464167 s, 88.2 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:59:48,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:59:48,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144414 s, 284 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 16:59:55,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 16:59:55,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182877 s, 224 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 16:59:58,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 16:59:58,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000750748 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:00:05,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:00:05,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132343 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:00:08,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:00:08,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786907 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:00:15,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:00:15,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812442 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:00:18,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:00:18,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204675 s, 200 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:00:25,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:00:25,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143374 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:00:28,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:00:28,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154277 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:00:35,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:00:35,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191274 s, 214 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:00:38,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:00:38,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00183154 s, 2.2 MB/s\n'; <rc> = 0
>Thread-14402::WARNING::2013-03-17 17:00:43,020::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:00:43,023::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:00:45,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:00:45,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766254 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:00:48,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:00:48,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780549 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:00:53,030::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:00:53,031::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:00:53,486::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:00:53,493::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:00:53,494::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:00:55,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:00:55,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123413 s, 332 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:00:58,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:00:58,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00779426 s, 526 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:01:06,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:01:06,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019987 s, 205 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:01:08,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:01:08,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179795 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:01:16,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:01:16,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815591 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:01:18,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:01:18,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150555 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:01:26,092::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:01:26,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0196843 s, 208 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:01:28,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:01:28,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185666 s, 221 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:01:36,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:01:36,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130087 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:01:38,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:01:39,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797838 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:01:46,199::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 17:01:46,200::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 17:01:46,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:01:46,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0216253 s, 94.7 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:01:46,272::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0216253 s, 94.7 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 17:01:46,272::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 17:01:46,273::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 17:01:46,274::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 17:01:46,275::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:01:46,281::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:01:46,281::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:01:46,282::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:01:46,282::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:01:46,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:01:46,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:01:46,710::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:01:46,711::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 17:01:46,714::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 17:01:46,715::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 17:01:46,716::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 17:01:46,717::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:01:47,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:01:47,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:01:47,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128843 s, 318 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:01:49,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:01:49,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00915144 s, 448 kB/s\n'; <rc> = 0
>Thread-14403::WARNING::2013-03-17 17:01:53,526::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:01:53,530::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:01:57,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:01:57,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862414 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:01:59,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:01:59,188::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000929631 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:02:03,536::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:02:03,537::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:02:04,020::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:02:04,026::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:02:04,028::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:02:07,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:02:07,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161589 s, 253 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:02:09,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:02:09,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014275 s, 287 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:02:17,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:02:17,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00223871 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:02:19,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:02:19,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152576 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:02:27,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:02:27,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149264 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:02:29,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:02:29,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173734 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:02:37,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:02:37,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167159 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:02:39,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:02:39,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00698091 s, 587 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:02:47,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:02:47,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172293 s, 238 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:02:49,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:02:49,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183485 s, 223 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:02:57,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:02:57,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166709 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:02:59,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:02:59,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123346 s, 3.3 MB/s\n'; <rc> = 0
>Thread-14404::WARNING::2013-03-17 17:03:04,065::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:03:04,071::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:03:07,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:03:07,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870279 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:03:09,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:03:09,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078894 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:03:14,074::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:03:14,075::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:03:14,523::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:03:14,530::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:03:14,531::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:03:17,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:03:17,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110308 s, 371 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:03:19,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:03:19,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157987 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:03:27,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:03:27,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00230888 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:03:29,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:03:29,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845866 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:03:37,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:03:37,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815422 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:03:39,984::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 17:03:39,986::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 17:03:39,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:03:40,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00130855 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:03:40,031::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00130855 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 17:03:40,032::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 17:03:40,032::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 17:03:40,034::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 17:03:40,035::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:03:40,035::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:03:40,036::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:03:40,036::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:03:40,037::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:03:40,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:03:40,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:03:40,466::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:03:40,468::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 17:03:40,470::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 17:03:40,471::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 17:03:40,472::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 17:03:40,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:03:40,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:03:40,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:03:40,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115221 s, 355 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:03:47,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:03:47,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155022 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:03:50,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:03:50,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150409 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:03:57,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:03:57,681::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141982 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:04:01,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:04:01,060::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758659 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:04:07,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:04:07,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855992 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:04:11,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:04:11,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142657 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14405::WARNING::2013-03-17 17:04:14,557::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:04:14,560::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:04:17,730::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:04:17,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881536 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:04:21,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:04:21,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100679 s, 4.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:04:24,573::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:04:24,575::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:04:25,024::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:04:25,030::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:04:25,032::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:04:27,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:04:27,818::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122609 s, 334 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:04:31,254::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:04:31,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837562 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:04:37,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:04:37,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110686 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:04:41,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:04:41,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786645 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:04:47,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:04:47,907::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161242 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:04:51,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:04:51,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117896 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:04:57,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:04:57,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146597 s, 279 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:05:01,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:05:01,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165476 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:05:07,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:05:07,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136376 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:05:11,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:05:11,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188009 s, 218 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:05:18,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:05:18,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156011 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:05:21,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:05:21,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821616 s, 5.0 MB/s\n'; <rc> = 0
>Thread-14406::WARNING::2013-03-17 17:05:25,061::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:05:25,066::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:05:28,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:05:28,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760369 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:05:31,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:05:31,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158476 s, 2.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:05:35,068::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 17:05:35,070::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:05:35,070::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:05:35,504::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:05:35,511::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:05:35,512::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:05:38,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:05:38,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138563 s, 296 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:05:41,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:05:41,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784211 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:05:48,171::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:05:48,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763205 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:05:51,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:05:51,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100574 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:05:58,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:05:58,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163374 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:06:01,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:06:02,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762675 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:06:08,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:06:08,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909693 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:06:12,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:06:12,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154455 s, 265 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:06:18,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:06:18,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124002 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:06:22,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:06:22,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147293 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:06:28,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:06:28,365::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767112 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:06:32,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:06:32,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154934 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14407::WARNING::2013-03-17 17:06:35,538::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:06:35,544::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:06:38,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:06:38,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810725 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:06:42,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:06:42,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821592 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:06:45,554::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:06:45,555::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:06:46,019::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:06:46,025::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:06:46,027::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:06:48,409::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 17:06:48,413::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 17:06:48,414::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:06:48,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0152787 s, 134 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:06:48,460::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0152787 s, 134 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 17:06:48,461::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 17:06:48,461::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 17:06:48,464::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 17:06:48,466::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:06:48,467::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:06:48,467::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:06:48,468::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:06:48,468::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:06:48,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:06:48,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:06:48,901::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:06:48,901::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 17:06:48,904::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 17:06:48,905::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 17:06:48,906::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 17:06:48,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:06:49,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:06:49,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:06:49,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00411293 s, 996 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:06:52,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:06:52,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146846 s, 279 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:06:59,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:06:59,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164381 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:07:02,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:07:02,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835919 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:07:09,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:07:09,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169705 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:07:12,537::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:07:12,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828625 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:07:19,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:07:19,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140053 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:07:22,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:07:22,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077726 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:07:29,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:07:29,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086658 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:07:32,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:07:32,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162638 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:07:39,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:07:39,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124354 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:07:42,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:07:42,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166477 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14408::WARNING::2013-03-17 17:07:46,055::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:07:46,058::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:07:49,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:07:49,611::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194686 s, 210 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:07:52,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:07:52,890::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076887 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:07:56,060::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:07:56,062::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:07:56,483::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:07:56,490::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:07:56,491::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:07:59,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:07:59,667::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0202571 s, 202 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:08:02,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:08:02,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805248 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:08:09,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:08:09,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799148 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:08:12,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:08:13,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831745 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:08:19,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:08:19,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0215665 s, 190 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:08:23,070::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:08:23,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135964 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:08:29,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:08:29,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163236 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:08:33,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:08:33,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149241 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:08:39,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:08:39,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132257 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:08:43,211::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 17:08:43,213::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 17:08:43,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:08:43,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00148936 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:08:43,259::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00148936 s, 1.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 17:08:43,260::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 17:08:43,260::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 17:08:43,262::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 17:08:43,263::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:08:43,263::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:08:43,264::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:08:43,264::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:08:43,265::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:08:43,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:08:43,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:08:43,709::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:08:43,710::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 17:08:43,713::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 17:08:43,714::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 17:08:43,715::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 17:08:43,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:08:44,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:08:44,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:08:44,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0065085 s, 629 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:08:49,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:08:49,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138121 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:08:54,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:08:54,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000955861 s, 4.3 MB/s\n'; <rc> = 0
>Thread-14409::WARNING::2013-03-17 17:08:56,521::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:08:56,526::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:08:59,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:08:59,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860797 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:09:04,260::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:09:04,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185312 s, 221 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:09:06,530::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:09:06,531::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:09:07,111::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:09:07,118::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:09:07,120::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:09:09,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:09:09,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150959 s, 271 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:09:14,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:09:14,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837921 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:09:20,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:09:20,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074549 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:09:24,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:09:24,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104575 s, 392 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:09:30,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:09:30,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089316 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:09:34,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:09:34,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813745 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:09:40,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:09:40,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000699955 s, 5.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:09:44,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:09:44,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833623 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:09:50,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:09:50,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00661668 s, 619 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:09:54,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:09:54,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784417 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:10:00,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:10:00,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168491 s, 243 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:10:04,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:10:04,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110655 s, 3.7 MB/s\n'; <rc> = 0
>Thread-14410::WARNING::2013-03-17 17:10:07,146::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:10:07,152::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:10:10,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:10:10,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845291 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:10:14,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:10:14,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101787 s, 4.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:10:17,154::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:10:17,155::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:10:17,637::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:10:17,644::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:10:17,645::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:10:20,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:10:20,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165621 s, 247 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:10:24,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:10:24,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923208 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:10:30,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:10:30,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789782 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:10:34,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:10:34,979::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00657933 s, 623 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:10:40,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:10:40,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143028 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:10:45,021::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:10:45,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167387 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:10:50,416::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:10:50,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144097 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:10:55,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:10:55,140::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893258 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:11:00,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:11:00,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148989 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:11:05,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:11:05,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189473 s, 216 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:11:10,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:11:10,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134843 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:11:15,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:11:15,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0207146 s, 198 kB/s\n'; <rc> = 0
>Thread-14411::WARNING::2013-03-17 17:11:17,673::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:11:17,677::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:11:20,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:11:20,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00653537 s, 627 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:11:25,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:11:25,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00373736 s, 1.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:11:27,680::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 17:11:27,681::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:11:27,682::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:11:28,144::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:11:28,150::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:11:28,152::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:11:30,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:11:30,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102099 s, 401 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:11:35,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:11:35,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197217 s, 208 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:11:40,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:11:40,805::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001244 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:11:45,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:11:45,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175203 s, 234 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:11:50,820::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 17:11:50,821::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 17:11:50,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:11:50,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00486282 s, 421 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:11:50,858::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00486282 s, 421 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 17:11:50,860::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 17:11:50,861::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 17:11:50,862::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 17:11:50,864::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:11:50,864::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:11:50,865::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:11:50,865::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:11:50,866::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:11:50,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:11:51,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:11:51,307::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:11:51,308::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 17:11:51,311::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 17:11:51,312::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 17:11:51,313::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 17:11:51,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:11:51,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:11:51,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:11:51,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159297 s, 257 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:11:55,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:11:55,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148057 s, 277 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:12:01,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:12:01,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779641 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:12:05,760::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:12:05,793::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147853 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:12:11,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:12:11,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00749236 s, 547 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:12:15,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:12:15,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137795 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:12:21,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:12:21,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170359 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:12:25,922::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:12:25,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000733714 s, 5.6 MB/s\n'; <rc> = 0
>Thread-14412::WARNING::2013-03-17 17:12:28,177::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:12:28,185::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:12:31,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:12:31,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008029 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:12:36,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:12:36,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821853 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:12:38,189::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:12:38,190::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:12:38,648::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:12:38,655::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:12:38,656::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:12:41,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:12:42,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166119 s, 247 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:12:46,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:12:46,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079972 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:12:52,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:12:52,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828348 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:12:56,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:12:56,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0201474 s, 203 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:13:02,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:13:02,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816155 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:13:06,260::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:13:06,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868781 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:13:12,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:13:12,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100896 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:13:16,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:13:16,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768662 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:13:22,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:13:22,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176447 s, 232 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:13:26,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:13:26,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144119 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:13:32,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:13:32,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00447181 s, 916 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:13:36,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:13:36,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894619 s, 4.6 MB/s\n'; <rc> = 0
>Thread-14413::WARNING::2013-03-17 17:13:38,681::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:13:38,684::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:13:42,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:13:42,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0200953 s, 204 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:13:46,564::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 17:13:46,565::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 17:13:46,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:13:46,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000789984 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:13:46,594::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000789984 s, 2.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 17:13:46,594::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 17:13:46,594::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 17:13:46,597::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 17:13:46,598::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:13:46,599::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:13:46,599::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:13:46,600::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:13:46,600::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:13:46,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:13:47,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:13:47,028::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:13:47,029::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 17:13:47,032::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 17:13:47,033::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 17:13:47,034::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 17:13:47,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:13:47,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:13:47,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:13:47,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152789 s, 268 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:13:48,694::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:13:48,695::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:13:49,178::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:13:49,185::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:13:49,186::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:13:52,292::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:13:52,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107902 s, 380 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:13:57,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:13:57,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854245 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:14:02,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:14:02,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810652 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:14:07,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:14:07,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0020756 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:14:12,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:14:12,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131632 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:14:17,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:14:17,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204147 s, 201 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:14:22,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:14:22,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754601 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:14:27,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:14:27,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897578 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:14:32,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:14:32,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154153 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:14:37,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:14:37,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00719033 s, 570 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:14:42,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:14:42,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104742 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:14:47,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:14:48,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203467 s, 201 kB/s\n'; <rc> = 0
>Thread-14414::WARNING::2013-03-17 17:14:49,215::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:14:49,217::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:14:52,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:14:52,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789801 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:14:58,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:14:58,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796018 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:14:59,220::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:14:59,221::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:14:59,659::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:14:59,666::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:14:59,667::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:15:02,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:15:02,681::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0222691 s, 184 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:15:08,524::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:15:08,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773591 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:15:12,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:15:12,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080348 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:15:18,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:15:18,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835142 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:15:22,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:15:22,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015816 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:15:28,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:15:28,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00373321 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:15:32,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:15:32,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013937 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:15:38,741::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:15:38,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000665336 s, 6.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:15:42,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:15:42,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135915 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:15:48,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:15:48,890::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834516 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:15:52,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:15:52,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780389 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:15:59,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:15:59,123::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000725391 s, 5.6 MB/s\n'; <rc> = 0
>Thread-14415::WARNING::2013-03-17 17:15:59,697::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:15:59,700::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:16:02,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:16:02,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018645 s, 220 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:16:09,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:16:09,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135889 s, 3.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:16:09,703::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:16:09,704::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:16:10,190::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:16:10,196::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:16:10,198::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:16:12,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:16:13,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115214 s, 356 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:16:19,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:16:19,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879443 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:16:23,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:16:23,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00294203 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:16:29,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:16:29,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817113 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:16:33,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:16:33,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850531 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:16:39,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:16:39,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078003 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:16:43,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:16:43,130::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915663 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:16:49,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:16:49,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016636 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:16:53,134::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 17:16:53,137::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 17:16:53,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:16:53,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00136822 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:16:53,174::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00136822 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 17:16:53,175::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 17:16:53,175::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 17:16:53,177::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 17:16:53,178::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:16:53,178::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:16:53,179::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:16:53,179::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:16:53,179::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:16:53,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:16:53,593::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:16:53,598::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:16:53,599::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 17:16:53,603::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 17:16:53,604::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 17:16:53,604::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 17:16:53,606::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:16:53,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:16:53,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:16:54,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166361 s, 246 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:16:59,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:16:59,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133919 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:17:04,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:17:04,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142844 s, 287 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:17:09,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:17:09,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00848529 s, 483 kB/s\n'; <rc> = 0
>Thread-14416::WARNING::2013-03-17 17:17:10,225::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:17:10,235::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:17:14,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:17:14,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000903263 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:17:20,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:17:20,295::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 17:17:20,297::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:17:20,298::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:17:20,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.028989 s, 141 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:17:20,753::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:17:20,758::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:17:20,760::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:17:24,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:17:24,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182437 s, 225 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:17:30,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:17:30,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101292 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:17:34,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:17:34,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000686609 s, 6.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:17:40,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:17:40,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126004 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:17:44,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:17:44,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105803 s, 387 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:17:50,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:17:50,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170649 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:17:54,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:17:54,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010417 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:18:00,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:18:00,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199113 s, 206 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:18:04,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:18:04,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0317945 s, 129 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:18:10,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:18:10,777::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185044 s, 221 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:18:14,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:18:14,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000999192 s, 4.1 MB/s\n'; <rc> = 0
>Thread-14417::WARNING::2013-03-17 17:18:20,787::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:18:20,793::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 17:18:20,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:18:20,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119444 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:18:24,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:18:24,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106142 s, 386 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:18:30,796::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:18:30,797::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:18:30,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:18:31,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121827 s, 336 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:18:31,300::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:18:31,306::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:18:31,307::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:18:34,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:18:34,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013015 s, 315 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:18:41,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:18:41,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911369 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:18:44,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:18:44,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017535 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:18:51,116::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 17:18:51,118::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 17:18:51,118::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:18:51,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000784722 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:18:51,151::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000784722 s, 2.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 17:18:51,151::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 17:18:51,152::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 17:18:51,153::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 17:18:51,154::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:18:51,155::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:18:51,155::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:18:51,156::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:18:51,156::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:18:51,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:18:51,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:18:51,596::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:18:51,597::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 17:18:51,599::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 17:18:51,600::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 17:18:51,601::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 17:18:51,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:18:52,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:18:52,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:18:52,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013852 s, 296 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:18:54,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:18:54,617::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120981 s, 339 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:19:02,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:19:02,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199102 s, 206 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:19:04,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:19:04,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748276 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:19:12,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:19:12,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199518 s, 205 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:19:14,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:19:14,700::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140647 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:19:22,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:19:22,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01789 s, 229 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:19:24,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:19:24,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169828 s, 2.4 MB/s\n'; <rc> = 0
>Thread-14418::WARNING::2013-03-17 17:19:31,337::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:19:31,343::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 17:19:32,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:19:32,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875983 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:19:34,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:19:34,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834385 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:19:41,355::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:19:41,357::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:19:41,749::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:19:41,756::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:19:41,757::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:19:42,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:19:42,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178916 s, 229 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:19:44,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:19:44,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187043 s, 219 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:19:52,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:19:52,569::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143531 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:19:54,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:19:54,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137025 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:20:02,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:20:02,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863357 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:20:04,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:20:04,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015758 s, 260 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:20:12,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:20:12,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816165 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:20:14,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:20:14,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199892 s, 205 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:20:22,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:20:22,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016561 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:20:25,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:20:25,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147093 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:20:32,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:20:32,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140417 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:20:35,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:20:35,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801976 s, 5.1 MB/s\n'; <rc> = 0
>Thread-14419::WARNING::2013-03-17 17:20:41,782::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:20:41,786::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 17:20:42,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:20:42,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778921 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:20:45,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:20:45,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168833 s, 2.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:20:51,794::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:20:51,795::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:20:52,313::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:20:52,319::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:20:52,321::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:20:52,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:20:53,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121346 s, 338 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:20:55,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:20:55,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132706 s, 309 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:21:03,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:21:03,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166573 s, 246 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:21:05,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:21:05,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170054 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:21:13,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:21:13,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0226051 s, 181 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:21:15,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:21:15,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00094066 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:21:23,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:21:23,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806264 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:21:25,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:21:25,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000972916 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:21:33,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:21:33,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161427 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:21:35,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:21:35,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761918 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:21:43,395::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:21:43,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823785 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:21:45,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:21:45,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121224 s, 3.4 MB/s\n'; <rc> = 0
>Thread-14420::WARNING::2013-03-17 17:21:52,345::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:21:52,349::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 17:21:53,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:21:53,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808664 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:21:55,401::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 17:21:55,403::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 17:21:55,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:21:55,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000868357 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:21:55,430::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000868357 s, 2.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 17:21:55,431::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 17:21:55,431::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 17:21:55,433::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 17:21:55,434::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:21:55,434::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:21:55,435::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:21:55,435::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:21:55,435::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:21:55,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:21:55,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:21:55,863::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:21:55,864::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 17:21:55,866::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 17:21:55,867::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 17:21:55,868::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 17:21:55,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:21:56,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:21:56,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:21:56,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153391 s, 267 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:22:02,351::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:22:02,353::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:22:02,866::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:22:02,873::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:22:02,874::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:22:03,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:22:03,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011323 s, 362 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:22:06,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:22:06,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122877 s, 333 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:22:13,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:22:13,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894641 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:22:16,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:22:16,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793507 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:22:23,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:22:23,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175234 s, 234 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:22:26,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:22:26,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811971 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:22:33,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:22:33,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101213 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:22:36,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:22:36,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144253 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:22:43,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:22:43,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171672 s, 239 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:22:46,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:22:46,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147315 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:22:53,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:22:54,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140992 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:22:56,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:22:56,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00288663 s, 1.4 MB/s\n'; <rc> = 0
>Thread-14421::WARNING::2013-03-17 17:23:02,901::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:23:02,905::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 17:23:04,070::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:23:04,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111831 s, 366 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:23:06,528::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:23:06,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000720115 s, 5.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:23:12,907::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 17:23:12,909::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:23:12,910::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:23:13,417::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:23:13,424::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:23:13,425::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:23:14,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:23:14,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138024 s, 297 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:23:16,560::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:23:16,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123516 s, 332 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:23:24,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:23:24,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000750304 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:23:26,605::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:23:26,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179034 s, 229 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:23:34,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:23:34,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101316 s, 404 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:23:36,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:23:36,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191783 s, 214 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:23:44,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:23:44,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00945544 s, 433 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:23:46,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:23:46,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174581 s, 235 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:23:54,482::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 17:23:54,483::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 17:23:54,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:23:54,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0150077 s, 136 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:23:54,544::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0150077 s, 136 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 17:23:54,545::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 17:23:54,546::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 17:23:54,547::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 17:23:54,548::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:23:54,549::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:23:54,549::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:23:54,550::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:23:54,550::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:23:54,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:23:54,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:23:54,982::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:23:54,983::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 17:23:54,985::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 17:23:54,986::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 17:23:54,987::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 17:23:54,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:23:55,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:23:55,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:23:55,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00586801 s, 698 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:23:56,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:23:56,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163273 s, 251 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:24:05,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:24:05,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838938 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:24:06,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:24:06,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136021 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14422::WARNING::2013-03-17 17:24:13,457::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:24:13,461::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 17:24:15,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:24:15,526::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801398 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:24:16,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:24:16,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106741 s, 3.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:24:23,471::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:24:23,472::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:24:23,922::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:24:23,929::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:24:23,930::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:24:25,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:24:25,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017716 s, 231 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:24:26,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:24:26,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014358 s, 285 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:24:35,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:24:35,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018043 s, 227 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:24:36,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:24:37,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799585 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:24:45,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:24:45,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793601 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:24:47,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:24:47,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814068 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:24:55,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:24:55,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019727 s, 208 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:24:57,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:24:57,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0222172 s, 184 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:25:05,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:25:05,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883545 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:25:07,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:25:07,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00322402 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:25:16,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:25:16,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816469 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:25:17,156::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:25:17,191::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123256 s, 3.3 MB/s\n'; <rc> = 0
>Thread-14423::WARNING::2013-03-17 17:25:23,955::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:25:23,961::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 17:25:26,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:25:26,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899746 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:25:27,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:25:27,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764851 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:25:33,964::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:25:33,966::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:25:34,439::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:25:34,445::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:25:34,447::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:25:36,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:25:36,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193205 s, 212 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:25:37,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:25:37,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192225 s, 213 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:25:46,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:25:46,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780071 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:25:47,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:25:47,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083969 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:25:56,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:25:56,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113691 s, 360 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:25:57,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:25:57,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000956623 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:26:06,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:26:06,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088409 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:26:07,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:26:07,404::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173279 s, 236 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:26:16,532::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:26:16,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00191539 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:26:17,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:26:17,439::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087147 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:26:26,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:26:26,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0218718 s, 187 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:26:27,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:26:27,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125513 s, 3.3 MB/s\n'; <rc> = 0
>Thread-14424::WARNING::2013-03-17 17:26:34,469::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:26:34,474::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 17:26:36,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:26:36,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085778 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:26:37,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:26:37,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821791 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:26:44,479::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:26:44,481::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:26:44,995::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:26:45,001::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:26:45,003::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:26:46,778::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:26:46,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114009 s, 359 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:26:47,530::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:26:47,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160628 s, 255 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:26:56,854::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:26:56,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841187 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:26:57,592::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 17:26:57,593::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 17:26:57,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:26:57,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00322288 s, 635 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:26:57,630::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00322288 s, 635 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 17:26:57,631::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 17:26:57,631::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 17:26:57,633::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 17:26:57,634::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:26:57,634::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:26:57,635::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:26:57,635::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:26:57,636::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:26:57,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:26:58,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:26:58,063::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:26:58,064::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 17:26:58,068::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 17:26:58,069::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 17:26:58,069::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 17:26:58,070::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:26:58,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:26:58,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:26:58,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140389 s, 292 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:27:06,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:27:06,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840767 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:27:08,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:27:08,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146463 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:27:17,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:27:17,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014893 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:27:18,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:27:18,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143248 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:27:27,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:27:27,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119143 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:27:28,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:27:28,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074809 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:27:37,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:27:37,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133859 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:27:38,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:27:38,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0215223 s, 190 kB/s\n'; <rc> = 0
>Thread-14425::WARNING::2013-03-17 17:27:45,030::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:27:45,033::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 17:27:47,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:27:47,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770944 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:27:48,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:27:48,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180842 s, 226 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:27:55,036::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:27:55,037::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:27:55,522::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:27:55,528::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:27:55,530::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:27:57,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:27:57,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0098911 s, 414 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:27:58,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:27:58,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171225 s, 239 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:28:07,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:28:07,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744008 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:28:08,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:28:08,837::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000979771 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:28:17,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:28:17,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804456 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:28:18,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:28:18,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160244 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:28:27,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:28:27,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176217 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:28:28,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:28:28,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134912 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:28:37,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:28:37,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159167 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:28:38,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:28:38,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139803 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:28:47,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:28:47,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00192117 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:28:48,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:28:49,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149591 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14426::WARNING::2013-03-17 17:28:55,564::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:28:55,568::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 17:28:57,809::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 17:28:57,810::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 17:28:57,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:28:57,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00085227 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:28:57,838::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00085227 s, 2.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 17:28:57,838::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 17:28:57,838::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 17:28:57,840::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 17:28:57,841::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:28:57,842::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:28:57,842::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:28:57,842::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:28:57,843::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:28:57,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:28:58,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:28:58,447::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:28:58,448::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 17:28:58,451::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 17:28:58,452::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 17:28:58,452::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 17:28:58,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:28:58,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:28:58,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:28:58,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0245075 s, 167 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:28:59,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:28:59,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100839 s, 406 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:29:05,573::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 17:29:05,574::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:29:05,575::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:29:06,014::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:29:06,020::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:29:06,022::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:29:08,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:29:08,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191644 s, 214 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:29:09,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:29:09,123::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164528 s, 249 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:29:18,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:29:19,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805517 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:29:19,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:29:19,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917674 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:29:29,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:29:29,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001706 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:29:29,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:29:29,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874796 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:29:39,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:29:39,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839252 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:29:39,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:29:39,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109592 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:29:49,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:29:49,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145186 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:29:49,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:29:49,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116114 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:29:59,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:29:59,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790661 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:29:59,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:29:59,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753901 s, 5.4 MB/s\n'; <rc> = 0
>Thread-14427::WARNING::2013-03-17 17:30:06,049::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:30:06,053::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 17:30:09,373::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:30:09,409::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838735 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:30:09,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:30:09,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088964 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:30:16,056::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:30:16,057::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:30:16,589::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:30:16,596::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:30:16,597::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:30:19,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:30:19,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116285 s, 352 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:30:19,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:30:19,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145374 s, 282 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:30:29,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:30:29,603::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:30:29,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011338 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:30:29,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148898 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:30:39,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:30:39,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778719 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:30:39,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:30:39,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00506969 s, 808 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:30:49,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:30:49,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777059 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:30:49,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:30:49,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000947272 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:30:59,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:30:59,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169612 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:30:59,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:30:59,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011329 s, 362 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:31:09,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:31:09,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141632 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:31:09,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:31:10,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0009583 s, 4.3 MB/s\n'; <rc> = 0
>Thread-14428::WARNING::2013-03-17 17:31:16,626::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:31:16,630::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:31:19,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:31:19,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757862 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:31:20,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:31:20,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144685 s, 2.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:31:26,632::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:31:26,633::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:31:27,082::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:31:27,089::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:31:27,090::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:31:29,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:31:29,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807738 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:31:30,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:31:30,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00805217 s, 509 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:31:39,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:31:39,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867986 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:31:40,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:31:40,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932245 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:31:49,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:31:49,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801162 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:31:50,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:31:50,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134253 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:31:59,998::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 17:31:59,999::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 17:32:00,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:32:00,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000858452 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:32:00,031::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000858452 s, 2.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 17:32:00,031::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 17:32:00,032::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 17:32:00,033::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 17:32:00,035::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:32:00,035::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:32:00,036::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:32:00,036::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:32:00,036::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:32:00,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:32:00,395::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:32:00,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0222663 s, 184 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:32:00,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:32:00,485::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:32:00,486::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 17:32:00,501::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 17:32:00,503::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 17:32:00,505::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 17:32:00,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:32:00,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:32:00,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:32:00,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118128 s, 347 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:32:10,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:32:10,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0216731 s, 189 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:32:10,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:32:10,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015339 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:32:20,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:32:20,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900507 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:32:21,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:32:21,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879758 s, 4.7 MB/s\n'; <rc> = 0
>Thread-14429::WARNING::2013-03-17 17:32:27,119::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:32:27,128::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 17:32:30,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:32:30,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786645 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:32:31,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:32:31,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867842 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:32:37,141::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:32:37,142::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:32:37,649::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:32:37,655::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:32:37,657::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:32:40,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:32:40,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194686 s, 210 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:32:41,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:32:41,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183102 s, 224 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:32:50,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:32:50,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143504 s, 285 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:32:51,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:32:51,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150647 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:33:00,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:33:01,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861436 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:33:01,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:33:01,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851762 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:33:11,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:33:11,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179244 s, 229 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:33:11,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:33:11,260::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157049 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:33:21,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:33:21,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160556 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:33:21,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:33:21,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00238094 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:33:31,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:33:31,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104668 s, 391 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:33:31,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:33:31,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807367 s, 5.1 MB/s\n'; <rc> = 0
>Thread-14430::WARNING::2013-03-17 17:33:37,687::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:33:37,691::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:33:41,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:33:41,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854468 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:33:41,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:33:41,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00275969 s, 1.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:33:47,694::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:33:47,695::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:33:48,153::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:33:48,160::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:33:48,161::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:33:51,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:33:51,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:33:51,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010979 s, 373 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:33:51,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133949 s, 306 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:34:01,571::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:34:01,614::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 17:34:01,618::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 17:34:01,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:34:01,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166545 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:34:01,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00143217 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:34:01,667::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00143217 s, 1.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 17:34:01,667::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 17:34:01,668::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 17:34:01,669::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 17:34:01,670::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:34:01,671::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:34:01,671::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:34:01,672::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:34:01,672::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:34:01,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:34:02,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:34:02,116::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:34:02,117::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 17:34:02,119::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 17:34:02,121::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 17:34:02,122::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 17:34:02,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:34:02,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:34:02,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:34:02,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.022559 s, 182 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:34:11,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:34:11,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145857 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:34:12,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:34:12,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129123 s, 317 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:34:21,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:34:21,734::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763001 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:34:22,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:34:22,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000735387 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:34:31,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:34:31,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821286 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:34:32,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:34:32,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00897423 s, 456 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:34:41,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:34:41,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127245 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:34:42,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:34:42,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135476 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14431::WARNING::2013-03-17 17:34:48,188::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:34:48,191::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:34:51,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:34:51,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152778 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:34:52,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:34:53,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181425 s, 226 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:34:58,193::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 17:34:58,194::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:34:58,195::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:34:58,700::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:34:58,707::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:34:58,708::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:35:01,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:35:01,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0095884 s, 427 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:35:03,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:35:03,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191211 s, 214 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:35:11,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:35:11,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797335 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:35:13,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:35:13,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116314 s, 352 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:35:21,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:35:21,979::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925237 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:35:23,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:35:23,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845431 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:35:31,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:35:32,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000935796 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:35:33,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:35:33,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159116 s, 257 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:35:42,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:35:42,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113279 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:35:43,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:35:43,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792747 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:35:52,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:35:52,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00461999 s, 887 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:35:53,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:35:53,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146484 s, 280 kB/s\n'; <rc> = 0
>Thread-14432::WARNING::2013-03-17 17:35:58,750::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:35:58,753::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:36:02,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:36:02,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849282 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:36:03,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:36:03,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0362482 s, 113 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:36:08,755::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:36:08,756::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:36:09,225::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:36:09,231::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:36:09,233::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:36:12,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:36:12,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017555 s, 233 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:36:13,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:36:13,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158761 s, 258 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:36:22,227::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:36:22,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00240406 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:36:23,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:36:23,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849588 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:36:32,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:36:32,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131589 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:36:33,814::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:36:33,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0195455 s, 210 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:36:42,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:36:42,331::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078254 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:36:43,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:36:43,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161938 s, 253 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:36:52,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:36:52,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148091 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:36:54,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:36:54,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887274 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:37:02,385::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 17:37:02,387::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 17:37:02,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:37:02,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00128607 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:37:02,427::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00128607 s, 1.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 17:37:02,428::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 17:37:02,428::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 17:37:02,429::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 17:37:02,432::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:37:02,433::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:37:02,433::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:37:02,434::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:37:02,434::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:37:02,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:37:02,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:37:02,875::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:37:02,876::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 17:37:02,879::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 17:37:02,880::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 17:37:02,880::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 17:37:02,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:37:03,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:37:03,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:37:03,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141016 s, 290 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:37:04,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:37:04,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0228947 s, 179 kB/s\n'; <rc> = 0
>Thread-14433::WARNING::2013-03-17 17:37:09,260::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:37:09,263::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:37:13,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:37:13,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798353 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:37:14,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:37:14,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194003 s, 211 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:37:19,266::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:37:19,268::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:37:19,750::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:37:19,756::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:37:19,758::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:37:23,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:37:23,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194171 s, 211 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:37:24,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:37:24,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104457 s, 392 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:37:33,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:37:33,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841645 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:37:34,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:37:34,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756041 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:37:43,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:37:43,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00099177 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:37:44,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:37:44,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0097013 s, 422 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:37:53,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:37:53,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148512 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:37:54,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:37:54,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148824 s, 275 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:38:03,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:38:03,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166115 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:38:04,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:38:04,658::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180215 s, 227 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:38:13,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:38:13,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880648 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:38:14,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:38:14,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113447 s, 361 kB/s\n'; <rc> = 0
>Thread-14434::WARNING::2013-03-17 17:38:19,784::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:38:19,787::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:38:23,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:38:23,593::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841641 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:38:24,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:38:24,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743 s, 5.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:38:29,789::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:38:29,790::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:38:30,248::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:38:30,255::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:38:30,256::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:38:33,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:38:33,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106079 s, 386 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:38:34,861::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:38:34,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018065 s, 227 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:38:43,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:38:43,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161127 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:38:44,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:38:44,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872837 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:38:53,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:38:53,739::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150713 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:38:55,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:38:55,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000959986 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:39:03,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:39:03,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00678674 s, 604 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:39:05,084::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 17:39:05,086::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 17:39:05,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:39:05,132::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0015274 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:39:05,133::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0015274 s, 1.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 17:39:05,134::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 17:39:05,135::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 17:39:05,137::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 17:39:05,138::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:39:05,138::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:39:05,139::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:39:05,139::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:39:05,139::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:39:05,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:39:05,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:39:05,583::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:39:05,584::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 17:39:05,586::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 17:39:05,588::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 17:39:05,588::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 17:39:05,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:39:05,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:39:05,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:39:05,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111012 s, 369 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:39:13,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:39:13,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932767 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:39:16,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:39:16,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151811 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:39:23,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:39:23,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00200745 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:39:26,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:39:26,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128696 s, 3.2 MB/s\n'; <rc> = 0
>Thread-14435::WARNING::2013-03-17 17:39:30,281::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:39:30,285::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:39:33,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:39:33,928::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0196737 s, 208 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:39:36,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:39:36,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769367 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:39:40,287::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:39:40,288::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:39:40,753::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:39:40,759::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:39:40,761::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:39:43,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:39:43,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110248 s, 372 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:39:46,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:39:46,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0330331 s, 124 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:39:53,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:39:54,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833751 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:39:56,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:39:56,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821365 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:40:04,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:40:04,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114278 s, 358 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:40:06,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:40:06,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145803 s, 281 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:40:14,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:40:14,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852591 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:40:16,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:40:16,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0057952 s, 707 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:40:24,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:40:24,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152081 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:40:26,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:40:26,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108999 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:40:34,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:40:34,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811808 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:40:36,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:40:36,700::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120897 s, 3.4 MB/s\n'; <rc> = 0
>Thread-14436::WARNING::2013-03-17 17:40:40,789::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:40:40,794::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:40:44,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:40:44,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013576 s, 302 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:40:46,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:40:46,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167184 s, 2.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:40:50,797::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 17:40:50,798::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:40:50,798::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:40:51,217::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:40:51,224::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:40:51,225::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:40:54,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:40:54,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181315 s, 226 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:40:56,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:40:56,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00980242 s, 418 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:41:04,324::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:41:04,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0274938 s, 149 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:41:06,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:41:06,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194193 s, 211 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:41:14,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:41:14,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001667 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:41:17,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:41:17,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786101 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:41:24,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:41:24,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000709555 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:41:27,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:41:27,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869775 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:41:34,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:41:34,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912356 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:41:37,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:41:37,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857613 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:41:44,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:41:44,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912839 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:41:47,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:41:47,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737425 s, 5.6 MB/s\n'; <rc> = 0
>Thread-14437::WARNING::2013-03-17 17:41:51,263::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:41:51,267::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:41:54,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:41:54,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00206648 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:41:57,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:41:57,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162926 s, 251 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:42:01,269::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:42:01,270::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:42:01,989::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:42:01,997::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:42:01,999::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:42:04,595::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 17:42:04,598::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 17:42:04,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:42:04,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0166462 s, 123 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:42:04,646::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0166462 s, 123 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 17:42:04,646::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 17:42:04,646::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 17:42:04,648::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 17:42:04,649::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:42:04,650::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:42:04,650::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:42:04,650::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:42:04,651::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:42:04,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:42:05,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:42:05,058::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:42:05,059::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 17:42:05,062::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 17:42:05,063::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 17:42:05,063::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 17:42:05,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:42:05,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:42:05,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:42:05,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118491 s, 346 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:42:07,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:42:07,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0348664 s, 117 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:42:15,492::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:42:15,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136132 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:42:17,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:42:17,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171133 s, 239 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:42:25,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:42:25,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0209592 s, 195 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:42:27,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:42:27,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881029 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:42:35,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:42:35,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140954 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:42:37,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:42:37,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172962 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:42:45,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:42:45,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0195094 s, 210 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:42:47,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:42:47,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00838837 s, 488 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:42:55,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:42:55,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078903 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:42:57,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:42:57,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136512 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14438::WARNING::2013-03-17 17:43:02,025::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:43:02,029::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:43:05,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:43:05,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0228468 s, 179 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:43:07,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:43:07,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152876 s, 268 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:43:12,031::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:43:12,032::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:43:12,469::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:43:12,475::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:43:12,477::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:43:15,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:43:15,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127471 s, 321 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:43:18,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:43:18,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188314 s, 218 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:43:25,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:43:25,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00253606 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:43:28,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:43:28,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105293 s, 389 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:43:35,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:43:35,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776911 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:43:38,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:43:38,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860754 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:43:45,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:43:45,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860592 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:43:48,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:43:48,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123664 s, 331 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:43:55,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:43:56,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814146 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:43:58,324::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:43:58,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138081 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:44:06,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:44:06,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00425949 s, 962 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:44:08,401::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 17:44:08,403::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 17:44:08,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:44:08,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0112824 s, 182 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:44:08,457::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0112824 s, 182 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 17:44:08,457::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 17:44:08,458::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 17:44:08,460::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 17:44:08,461::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:44:08,461::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:44:08,462::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:44:08,462::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:44:08,464::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:44:08,465::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:44:08,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:44:08,951::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:44:08,952::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 17:44:08,954::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 17:44:08,956::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 17:44:08,957::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 17:44:08,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:44:09,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:44:09,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:44:09,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129323 s, 317 kB/s\n'; <rc> = 0
>Thread-14439::WARNING::2013-03-17 17:44:12,502::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:44:12,509::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:44:16,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:44:16,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826474 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:44:19,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:44:19,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891303 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:44:22,517::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:44:22,518::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:44:23,012::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:44:23,019::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:44:23,020::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:44:26,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:44:26,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00667227 s, 614 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:44:29,490::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:44:29,526::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011169 s, 367 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:44:36,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:44:36,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846943 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:44:39,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:44:39,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817629 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:44:46,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:44:46,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000955941 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:44:49,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:44:49,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158903 s, 258 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:44:56,209::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:44:56,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148774 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:44:59,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:44:59,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117616 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:45:06,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:45:06,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019654 s, 208 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:45:09,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:45:09,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204692 s, 200 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:45:16,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:45:16,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117576 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:45:19,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:45:19,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134095 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14440::WARNING::2013-03-17 17:45:23,046::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:45:23,049::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:45:26,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:45:26,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00994515 s, 412 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:45:29,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:45:30,017::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.008185 s, 500 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:45:33,052::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:45:33,053::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:45:33,487::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:45:33,494::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:45:33,495::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:45:36,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:45:36,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145303 s, 282 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:45:40,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:45:40,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00922662 s, 444 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:45:46,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:45:46,490::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821384 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:45:50,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:45:50,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120321 s, 340 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:45:56,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:45:56,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172595 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:46:00,230::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:46:00,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153736 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:46:06,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:46:06,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142855 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:46:10,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:46:10,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00905095 s, 453 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:46:16,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:46:16,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124427 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:46:20,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:46:20,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751775 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:46:26,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:46:26,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793227 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:46:30,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:46:30,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0019345 s, 2.1 MB/s\n'; <rc> = 0
>Thread-14441::WARNING::2013-03-17 17:46:33,524::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:46:33,528::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:46:36,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:46:36,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790601 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:46:40,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:46:40,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783246 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:46:43,530::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 17:46:43,531::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:46:43,531::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:46:43,984::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:46:43,991::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:46:43,992::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:46:46,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:46:46,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194534 s, 211 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:46:50,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:46:50,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184587 s, 222 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:46:56,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:46:56,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0224496 s, 182 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:47:00,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:47:00,793::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193469 s, 212 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:47:06,845::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 17:47:06,846::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 17:47:06,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:47:06,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0203026 s, 101 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:47:06,896::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0203026 s, 101 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 17:47:06,897::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 17:47:06,897::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 17:47:06,900::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 17:47:06,901::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:47:06,902::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:47:06,902::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:47:06,903::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:47:06,903::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:47:06,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:47:07,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:47:07,365::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:47:07,366::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 17:47:07,368::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 17:47:07,369::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 17:47:07,370::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 17:47:07,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:47:07,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:47:07,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:47:07,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131301 s, 312 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:47:10,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:47:10,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00964008 s, 425 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:47:17,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:47:17,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132588 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:47:20,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:47:20,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.020172 s, 203 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:47:27,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:47:27,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162601 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:47:31,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:47:31,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166492 s, 246 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:47:37,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:47:37,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157579 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:47:41,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:47:41,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008396 s, 4.9 MB/s\n'; <rc> = 0
>Thread-14442::WARNING::2013-03-17 17:47:44,022::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:47:44,024::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:47:47,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:47:47,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876019 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:47:51,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:47:51,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0327629 s, 125 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:47:54,032::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:47:54,033::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:47:54,513::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:47:54,520::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:47:54,521::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:47:58,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:47:58,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143988 s, 284 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:48:01,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:48:01,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138617 s, 295 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:48:08,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:48:08,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181087 s, 226 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:48:11,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:48:11,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127987 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:48:18,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:48:18,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773334 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:48:21,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:48:21,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811237 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:48:28,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:48:28,195::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176817 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:48:31,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:48:31,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154765 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:48:38,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:48:38,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132581 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:48:41,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:48:41,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017105 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:48:48,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:48:48,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138876 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:48:51,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:48:51,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000979785 s, 4.2 MB/s\n'; <rc> = 0
>Thread-14443::WARNING::2013-03-17 17:48:54,547::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:48:54,551::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:48:58,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:48:58,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000968376 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:49:01,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:49:01,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893223 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:49:04,553::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:49:04,554::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:49:05,026::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:49:05,033::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:49:05,034::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:49:08,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:49:08,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0205367 s, 199 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:49:11,918::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 17:49:11,920::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 17:49:11,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:49:11,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.014872 s, 138 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:49:11,964::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.014872 s, 138 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 17:49:11,965::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 17:49:11,965::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 17:49:11,966::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 17:49:11,968::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:49:11,968::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:49:11,969::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:49:11,969::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:49:11,969::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:49:11,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:49:12,392::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:49:12,396::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:49:12,397::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 17:49:12,401::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 17:49:12,402::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 17:49:12,402::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 17:49:12,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:49:12,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:49:12,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:49:12,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107552 s, 381 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:49:18,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:49:18,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137188 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:49:22,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:49:22,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077805 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:49:28,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:49:28,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897536 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:49:32,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:49:33,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826892 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:49:38,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:49:38,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861571 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:49:43,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:49:43,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148106 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:49:48,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:49:48,552::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141971 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:49:53,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:49:53,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806466 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:49:58,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:49:58,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082221 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:50:03,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:50:03,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121803 s, 3.4 MB/s\n'; <rc> = 0
>Thread-14444::WARNING::2013-03-17 17:50:05,060::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:50:05,063::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:50:08,596::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:50:08,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001047 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:50:13,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:50:13,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158216 s, 2.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:50:15,066::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:50:15,067::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:50:15,482::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:50:15,489::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:50:15,490::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:50:18,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:50:18,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018474 s, 222 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:50:23,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:50:23,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019936 s, 205 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:50:28,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:50:28,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120159 s, 341 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:50:33,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:50:33,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127195 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:50:38,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:50:38,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885765 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:50:43,571::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:50:43,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858199 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:50:48,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:50:48,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000970615 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:50:53,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:50:53,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143775 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:50:58,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:50:58,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00457244 s, 896 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:51:03,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:51:03,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152346 s, 269 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:51:08,854::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:51:08,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000951148 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:51:13,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:51:13,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110022 s, 3.7 MB/s\n'; <rc> = 0
>Thread-14445::WARNING::2013-03-17 17:51:15,519::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:51:15,526::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:51:18,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:51:18,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885716 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:51:23,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:51:23,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785213 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:51:25,529::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:51:25,530::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:51:25,981::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:51:25,988::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:51:25,989::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:51:28,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:51:28,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000972676 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:51:33,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:51:34,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00963113 s, 425 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:51:38,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:51:39,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206558 s, 198 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:51:44,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:51:44,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187135 s, 219 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:51:49,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:51:49,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.020809 s, 197 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:51:54,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:51:54,238::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134717 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:51:59,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:51:59,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850359 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:52:04,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:52:04,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141922 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:52:09,131::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 17:52:09,132::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 17:52:09,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:52:09,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0221935 s, 92.3 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:52:09,190::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0221935 s, 92.3 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 17:52:09,190::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 17:52:09,191::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 17:52:09,192::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 17:52:09,193::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:52:09,194::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:52:09,195::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:52:09,195::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:52:09,195::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:52:09,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:52:09,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:52:09,632::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:52:09,633::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 17:52:09,636::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 17:52:09,637::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 17:52:09,637::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 17:52:09,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:52:10,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:52:10,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:52:10,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00895415 s, 457 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:52:14,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:52:14,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184173 s, 222 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:52:20,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:52:20,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0216709 s, 189 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:52:24,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:52:24,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111661 s, 3.7 MB/s\n'; <rc> = 0
>Thread-14446::WARNING::2013-03-17 17:52:26,020::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:52:26,026::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:52:30,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:52:30,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000708742 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:52:34,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:52:34,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091325 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:52:36,029::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 17:52:36,030::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:52:36,031::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:52:36,514::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:52:36,520::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:52:36,522::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:52:40,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:52:40,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117832 s, 348 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:52:44,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:52:44,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144272 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:52:50,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:52:50,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868842 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:52:54,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:52:54,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151658 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:53:00,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:53:00,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000715769 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:53:04,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:53:04,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133323 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:53:10,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:53:10,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171864 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:53:14,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:53:14,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880196 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:53:20,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:53:20,392::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.02032 s, 202 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:53:24,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:53:24,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799761 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:53:30,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:53:30,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851788 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:53:35,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:53:35,060::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116058 s, 3.5 MB/s\n'; <rc> = 0
>Thread-14447::WARNING::2013-03-17 17:53:36,554::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:53:36,556::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:53:40,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:53:40,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000937466 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:53:45,118::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:53:45,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174734 s, 234 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:53:46,561::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:53:46,563::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:53:47,061::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:53:47,068::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:53:47,070::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:53:50,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:53:50,512::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00984308 s, 416 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:53:55,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:53:55,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153738 s, 266 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:54:00,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:54:00,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0216717 s, 189 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:54:05,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:54:05,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00713959 s, 574 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:54:10,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:54:10,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157259 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:54:15,360::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 17:54:15,362::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 17:54:15,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:54:15,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000690629 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:54:15,391::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000690629 s, 3.0 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 17:54:15,391::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 17:54:15,391::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 17:54:15,393::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 17:54:15,394::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:54:15,395::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:54:15,397::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:54:15,398::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:54:15,398::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:54:15,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:54:15,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:54:15,832::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:54:15,833::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 17:54:15,836::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 17:54:15,837::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 17:54:15,838::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 17:54:15,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:54:16,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:54:16,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:54:16,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00533166 s, 768 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:54:20,625::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:54:20,671::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00682336 s, 600 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:54:26,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:54:26,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00839643 s, 488 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:54:30,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:54:30,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0217009 s, 189 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:54:36,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:54:36,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825981 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:54:40,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:54:40,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011374 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:54:46,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:54:46,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103504 s, 396 kB/s\n'; <rc> = 0
>Thread-14448::WARNING::2013-03-17 17:54:47,096::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:54:47,100::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:54:50,781::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:54:50,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176573 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:54:56,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:54:56,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000727758 s, 5.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:54:57,115::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:54:57,116::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:54:57,579::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:54:57,586::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:54:57,587::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:55:00,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:55:00,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181925 s, 225 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:55:07,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:55:07,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177746 s, 230 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:55:10,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:55:10,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000947241 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:55:17,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:55:17,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816082 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:55:20,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:55:20,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767902 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:55:27,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:55:27,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.02235 s, 183 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:55:30,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:55:30,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151802 s, 270 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:55:37,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:55:37,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152892 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:55:41,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:55:41,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152476 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:55:47,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:55:47,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000959886 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:55:51,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:55:51,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00098102 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:55:57,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:55:57,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773261 s, 5.3 MB/s\n'; <rc> = 0
>Thread-14449::WARNING::2013-03-17 17:55:57,615::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:55:57,624::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:56:01,092::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:56:01,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778221 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:56:07,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:56:07,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000953461 s, 4.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:56:07,639::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:56:07,753::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:56:08,259::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:56:08,267::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:56:08,282::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:56:11,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:56:11,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103133 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:56:17,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:56:17,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769133 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:56:21,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:56:21,266::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0213224 s, 192 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:56:27,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:56:27,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133974 s, 306 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:56:31,271::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:56:31,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154742 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:56:37,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:56:38,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880344 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:56:41,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:56:41,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808728 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:56:48,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:56:48,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147243 s, 278 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:56:51,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:56:51,399::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161787 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:56:58,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:56:58,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153139 s, 267 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:57:01,405::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:57:01,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000954158 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:57:08,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:57:08,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757182 s, 5.4 MB/s\n'; <rc> = 0
>Thread-14450::WARNING::2013-03-17 17:57:08,337::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:57:08,340::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:57:11,441::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 17:57:11,442::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 17:57:11,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:57:11,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00118961 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:57:11,478::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00118961 s, 1.7 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 17:57:11,478::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 17:57:11,479::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 17:57:11,480::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 17:57:11,481::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:57:11,482::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:57:11,482::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:57:11,482::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:57:11,483::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 17:57:11,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:57:11,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:57:11,882::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:57:11,883::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 17:57:11,885::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 17:57:11,886::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 17:57:11,887::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 17:57:11,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:57:12,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:57:12,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:57:12,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0248285 s, 165 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:57:18,342::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:57:18,343::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:57:18,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:57:18,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159952 s, 256 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:57:18,975::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:57:18,981::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:57:18,982::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:57:22,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:57:22,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812501 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:57:28,530::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:57:28,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753265 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:57:32,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:57:32,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800733 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:57:38,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:57:38,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141531 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:57:42,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:57:42,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755046 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:57:48,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:57:48,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917106 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:57:52,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:57:52,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866498 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:57:58,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:57:58,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000710965 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:58:02,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:58:02,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741463 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:58:08,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:58:08,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863147 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:58:12,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:58:12,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728511 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:58:18,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:58:18,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168792 s, 2.4 MB/s\n'; <rc> = 0
>Thread-14451::WARNING::2013-03-17 17:58:19,017::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:58:19,026::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 17:58:22,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:58:22,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088154 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:58:29,040::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 17:58:29,042::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:58:29,043::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:58:29,079::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:58:29,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00926406 s, 442 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:58:29,574::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:58:29,579::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:58:29,581::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 17:58:32,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:58:32,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013489 s, 304 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:58:39,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:58:39,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000975275 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:58:42,648::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:58:42,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0205182 s, 200 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:58:49,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:58:49,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00749151 s, 547 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:58:52,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:58:52,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754212 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:58:59,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:58:59,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760194 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:59:02,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:59:02,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187158 s, 219 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:59:09,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:59:09,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175452 s, 233 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:59:12,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:59:12,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779497 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:59:19,578::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 17:59:19,579::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 17:59:19,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:59:19,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00149382 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:59:19,620::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00149382 s, 1.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 17:59:19,620::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 17:59:19,620::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 17:59:19,623::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 17:59:19,624::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:59:19,624::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:59:19,625::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:59:19,625::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:59:19,625::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 17:59:19,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:59:20,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:59:20,132::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:59:20,134::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 17:59:20,137::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 17:59:20,138::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 17:59:20,138::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 17:59:20,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:59:20,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:59:20,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:59:20,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190252 s, 215 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:59:22,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:59:22,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107375 s, 381 kB/s\n'; <rc> = 0
>Thread-14452::WARNING::2013-03-17 17:59:29,624::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 17:59:29,628::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 17:59:30,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:59:30,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826725 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:59:32,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:59:32,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867719 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 17:59:39,631::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 17:59:39,632::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 17:59:40,069::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 17:59:40,075::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 17:59:40,076::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 17:59:40,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:59:40,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143964 s, 285 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:59:42,941::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:59:42,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181462 s, 226 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 17:59:50,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 17:59:50,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802633 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 17:59:52,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 17:59:53,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156854 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:00:00,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:00:00,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176773 s, 232 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:00:03,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:00:03,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833354 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:00:10,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:00:11,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117755 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:00:13,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:00:13,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143702 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:00:21,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:00:21,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000975287 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:00:23,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:00:23,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078057 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:00:31,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:00:31,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142875 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:00:33,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:00:33,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126079 s, 325 kB/s\n'; <rc> = 0
>Thread-14453::WARNING::2013-03-17 18:00:40,103::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:00:40,110::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:00:41,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:00:41,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00097075 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:00:43,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:00:43,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079596 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:00:50,121::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:00:50,122::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:00:50,619::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:00:50,628::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:00:50,630::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:00:51,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:00:51,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00638406 s, 642 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:00:53,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:00:53,309::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181668 s, 225 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:01:01,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:01:01,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000704335 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:01:03,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:01:03,347::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849206 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:01:11,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:01:11,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129203 s, 317 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:01:13,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:01:13,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000708737 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:01:21,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:01:21,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844209 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:01:23,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:01:23,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140806 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:01:31,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:01:31,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167297 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:01:33,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:01:33,483::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142303 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:01:41,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:01:41,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103635 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:01:43,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:01:43,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0225447 s, 182 kB/s\n'; <rc> = 0
>Thread-14454::WARNING::2013-03-17 18:01:50,659::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:01:50,664::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:01:51,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:01:51,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874709 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:01:53,556::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:01:53,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896306 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:02:00,669::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:02:00,671::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:02:01,184::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:02:01,192::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:02:01,198::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:02:01,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:02:01,881::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0224398 s, 183 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:02:03,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:02:03,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179808 s, 228 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:02:11,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:02:11,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139481 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:02:13,638::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 18:02:13,640::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 18:02:13,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:02:13,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00127336 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:02:13,681::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00127336 s, 1.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 18:02:13,682::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 18:02:13,682::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 18:02:13,684::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 18:02:13,685::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:02:13,686::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:02:13,686::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:02:13,687::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:02:13,687::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:02:13,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:02:14,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:02:14,132::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:02:14,133::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 18:02:14,135::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 18:02:14,136::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 18:02:14,137::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 18:02:14,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:02:14,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:02:14,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:02:14,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163754 s, 250 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:02:22,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:02:22,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00827637 s, 495 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:02:24,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:02:24,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919515 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:02:32,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:02:32,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744755 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:02:34,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:02:34,631::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152485 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:02:42,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:02:42,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101581 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:02:44,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:02:44,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820878 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:02:52,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:02:52,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144334 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:02:54,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:02:54,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912209 s, 4.5 MB/s\n'; <rc> = 0
>Thread-14455::WARNING::2013-03-17 18:03:01,230::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:03:01,232::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:03:02,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:03:02,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180325 s, 227 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:03:04,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:03:04,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829446 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:03:11,235::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:03:11,236::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:03:11,787::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:03:11,793::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:03:11,794::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:03:12,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:03:12,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174228 s, 235 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:03:14,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:03:14,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188731 s, 217 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:03:22,537::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:03:22,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866416 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:03:24,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:03:24,881::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0224622 s, 182 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:03:32,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:03:32,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141561 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:03:34,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:03:34,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0220743 s, 186 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:03:42,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:03:42,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013697 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:03:44,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:03:44,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142621 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:03:52,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:03:52,819::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145913 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:03:54,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:03:55,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146823 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:04:02,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:04:02,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753675 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:04:05,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:04:05,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175584 s, 2.3 MB/s\n'; <rc> = 0
>Thread-14456::WARNING::2013-03-17 18:04:11,819::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:04:11,823::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:04:12,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:04:12,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00368524 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:04:15,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:04:15,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000703162 s, 5.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:04:21,831::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 18:04:21,832::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:04:21,833::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:04:22,307::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:04:22,314::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:04:22,315::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:04:23,000::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 18:04:23,004::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 18:04:23,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:04:23,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0130118 s, 157 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:04:23,043::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0130118 s, 157 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 18:04:23,044::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 18:04:23,044::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 18:04:23,046::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 18:04:23,047::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:04:23,047::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:04:23,048::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:04:23,048::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:04:23,049::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:04:23,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:04:23,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:04:23,446::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:04:23,447::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 18:04:23,449::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 18:04:23,450::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 18:04:23,451::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 18:04:23,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:04:23,805::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:04:23,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:04:23,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113718 s, 360 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:04:25,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:04:25,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140807 s, 291 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:04:33,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:04:33,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00683755 s, 599 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:04:35,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:04:35,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883446 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:04:43,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:04:43,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00222758 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:04:45,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:04:45,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138399 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:04:54,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:04:54,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774404 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:04:55,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:04:55,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014335 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:05:04,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:05:04,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826164 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:05:05,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:05:05,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.022967 s, 178 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:05:14,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:05:14,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014821 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:05:15,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:05:15,399::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149865 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14457::WARNING::2013-03-17 18:05:22,342::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:05:22,345::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:05:24,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:05:24,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166519 s, 246 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:05:25,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:05:25,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817281 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:05:32,347::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:05:32,349::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:05:32,831::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:05:32,838::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:05:32,839::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:05:34,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:05:34,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00775243 s, 528 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:05:35,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:05:35,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0277424 s, 148 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:05:44,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:05:44,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162591 s, 252 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:05:45,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:05:45,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825565 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:05:54,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:05:54,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763662 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:05:55,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:05:55,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154957 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:06:04,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:06:04,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162207 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:06:05,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:06:05,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145337 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:06:14,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:06:14,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125538 s, 326 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:06:15,648::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:06:15,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00330008 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:06:24,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:06:24,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140978 s, 291 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:06:25,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:06:25,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147726 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14458::WARNING::2013-03-17 18:06:32,866::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:06:32,870::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:06:34,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:06:34,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153112 s, 268 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:06:35,742::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:06:35,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077095 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:06:42,872::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:06:42,873::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:06:43,308::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:06:43,314::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:06:43,315::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:06:44,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:06:44,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103901 s, 394 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:06:45,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:06:45,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170806 s, 240 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:06:55,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:06:55,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787785 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:06:55,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:06:55,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843035 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:07:05,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:07:05,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151261 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:07:05,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:07:05,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832821 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:07:15,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:07:15,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784035 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:07:15,887::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 18:07:15,889::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 18:07:15,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:07:15,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000861269 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:07:15,924::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000861269 s, 2.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 18:07:15,925::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 18:07:15,925::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 18:07:15,928::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 18:07:15,929::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:07:15,930::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:07:15,930::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:07:15,931::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:07:15,932::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:07:15,933::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:07:16,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:07:16,388::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:07:16,389::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 18:07:16,393::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 18:07:16,394::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 18:07:16,395::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 18:07:16,396::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:07:16,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:07:16,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:07:16,820::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00925693 s, 442 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:07:25,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:07:25,311::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136817 s, 299 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:07:26,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:07:26,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0220427 s, 186 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:07:35,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:07:35,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142744 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:07:36,894::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:07:36,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0196595 s, 208 kB/s\n'; <rc> = 0
>Thread-14459::WARNING::2013-03-17 18:07:43,340::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:07:43,343::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:07:45,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:07:45,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00675866 s, 606 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:07:46,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:07:46,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00093155 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:07:53,374::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:07:53,375::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:07:53,882::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:07:53,888::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:07:53,890::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:07:55,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:07:55,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00917471 s, 446 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:07:56,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:07:57,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117935 s, 347 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:08:05,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:08:05,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109741 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:08:07,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:08:07,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00256479 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:08:15,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:08:15,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772316 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:08:17,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:08:17,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001441 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:08:25,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:08:25,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744107 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:08:27,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:08:27,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102582 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:08:35,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:08:35,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859516 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:08:37,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:08:37,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016547 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:08:45,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:08:45,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839323 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:08:47,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:08:47,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932022 s, 4.4 MB/s\n'; <rc> = 0
>Thread-14460::WARNING::2013-03-17 18:08:53,927::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:08:53,931::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:08:55,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:08:55,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790258 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:08:57,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:08:57,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000988977 s, 4.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:09:03,933::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:09:03,934::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:09:04,425::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:09:04,431::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:09:04,433::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:09:06,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:09:06,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118868 s, 345 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:09:07,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:09:07,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119791 s, 342 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:09:16,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:09:16,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144351 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:09:17,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:09:17,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000998397 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:09:26,182::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 18:09:26,183::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 18:09:26,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:09:26,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000895121 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:09:26,209::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000895121 s, 2.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 18:09:26,210::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 18:09:26,210::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 18:09:26,212::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 18:09:26,213::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:09:26,214::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:09:26,214::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:09:26,215::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:09:26,215::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:09:26,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:09:26,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:09:26,649::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:09:26,650::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 18:09:26,652::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 18:09:26,653::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 18:09:26,654::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 18:09:26,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:09:27,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:09:27,014::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:09:27,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00962118 s, 426 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:09:27,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:09:27,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162342 s, 252 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:09:37,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:09:37,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124237 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:09:37,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:09:37,450::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146691 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:09:47,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:09:47,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166569 s, 246 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:09:47,455::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:09:47,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803811 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:09:57,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:09:57,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810884 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:09:57,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:09:57,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135541 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14461::WARNING::2013-03-17 18:10:04,458::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:10:04,462::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:10:07,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:10:07,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170192 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:10:07,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:10:07,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790396 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:10:14,467::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 18:10:14,468::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:10:14,469::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:10:14,893::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:10:14,899::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:10:14,901::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:10:17,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:10:17,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00628958 s, 651 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:10:17,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:10:17,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204879 s, 200 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:10:27,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:10:27,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825461 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:10:27,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:10:27,756::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00995502 s, 411 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:10:37,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:10:37,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078922 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:10:37,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:10:37,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00440291 s, 930 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:10:47,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:10:47,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015907 s, 257 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:10:47,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:10:47,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000926636 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:10:57,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:10:57,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135945 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:10:57,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:10:57,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000720227 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:11:07,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:11:07,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0312964 s, 131 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:11:07,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:11:07,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104442 s, 3.9 MB/s\n'; <rc> = 0
>Thread-14462::WARNING::2013-03-17 18:11:14,929::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:11:14,932::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 18:11:17,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:11:18,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:11:18,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00374058 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:11:18,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125703 s, 326 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:11:24,936::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:11:24,937::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:11:25,331::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:11:25,338::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:11:25,340::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:11:28,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:11:28,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:11:28,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871452 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:11:28,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182162 s, 225 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:11:38,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:11:38,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:11:38,364::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108841 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:11:38,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000961318 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:11:48,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:11:48,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:11:48,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00381288 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:11:48,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900653 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:11:58,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:11:58,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:11:58,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114806 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:11:58,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168474 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:12:08,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:12:08,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:12:08,700::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117356 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:12:08,734::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199521 s, 205 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:12:18,749::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 18:12:18,750::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 18:12:18,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:12:18,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00150418 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:12:18,794::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00150418 s, 1.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 18:12:18,794::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 18:12:18,794::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 18:12:18,797::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 18:12:18,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:12:18,835::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:12:18,858::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:12:18,864::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:12:18,866::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:12:18,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112594 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:12:18,879::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:12:18,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:12:19,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:12:19,327::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:12:19,328::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 18:12:19,331::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 18:12:19,332::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 18:12:19,332::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 18:12:19,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:12:19,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:12:19,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:12:19,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0496526 s, 82.5 kB/s\n'; <rc> = 0
>Thread-14463::WARNING::2013-03-17 18:12:25,365::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:12:25,370::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:12:28,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:12:29,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00907251 s, 451 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:12:29,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:12:29,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881054 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:12:35,372::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:12:35,373::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:12:35,868::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:12:35,875::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:12:35,877::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:12:39,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:12:39,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118976 s, 344 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:12:39,899::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:12:39,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145325 s, 282 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:12:49,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:12:49,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782409 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:12:49,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:12:49,973::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000735203 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:12:59,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:12:59,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150639 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:12:59,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:13:00,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860312 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:13:09,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:13:09,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136332 s, 300 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:13:10,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:13:10,054::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014212 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:13:19,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:13:19,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860292 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:13:20,060::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:13:20,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151183 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:13:29,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:13:29,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00997618 s, 411 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:13:30,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:13:30,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0022924 s, 1.8 MB/s\n'; <rc> = 0
>Thread-14464::WARNING::2013-03-17 18:13:35,900::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:13:35,904::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:13:39,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:13:39,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010245 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:13:40,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:13:40,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799439 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:13:45,907::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:13:45,909::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:13:46,313::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:13:46,320::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:13:46,321::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:13:49,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:13:49,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00649246 s, 631 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:13:50,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:13:50,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114245 s, 359 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:13:59,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:13:59,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000750673 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:14:00,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:14:00,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126945 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:14:09,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:14:09,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154986 s, 264 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:14:10,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:14:10,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00309019 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:14:19,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:14:19,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103012 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:14:20,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:14:20,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161036 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:14:29,880::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 18:14:29,881::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 18:14:29,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:14:29,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0101054 s, 203 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:14:29,925::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0101054 s, 203 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 18:14:29,926::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 18:14:29,926::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 18:14:29,928::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 18:14:29,928::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:14:29,929::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:14:29,930::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:14:29,930::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:14:29,930::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:14:29,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:14:30,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:14:30,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:14:30,418::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:14:30,419::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 18:14:30,421::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 18:14:30,423::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 18:14:30,423::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 18:14:30,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:14:30,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203193 s, 202 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:14:30,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:14:30,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:14:30,880::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127156 s, 322 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:14:40,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:14:40,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173285 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:14:40,922::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:14:40,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763054 s, 5.4 MB/s\n'; <rc> = 0
>Thread-14465::WARNING::2013-03-17 18:14:46,349::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:14:46,357::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 18:14:50,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:14:50,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158803 s, 258 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:14:50,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:14:51,040::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158868 s, 2.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:14:56,369::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:14:56,370::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:14:56,852::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:14:56,858::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:14:56,860::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:15:00,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:15:00,617::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186113 s, 220 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:15:01,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:15:01,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882138 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:15:10,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:15:10,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147724 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:15:11,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:15:11,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127247 s, 322 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:15:20,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:15:20,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137288 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:15:21,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:15:21,297::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151126 s, 271 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:15:30,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:15:30,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000927393 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:15:31,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:15:31,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141327 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:15:40,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:15:40,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00569254 s, 720 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:15:41,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:15:41,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134648 s, 304 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:15:50,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:15:50,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140934 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:15:51,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:15:51,591::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00667732 s, 613 kB/s\n'; <rc> = 0
>Thread-14466::WARNING::2013-03-17 18:15:56,897::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:15:56,904::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 18:16:00,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:16:01,017::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145793 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:16:01,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:16:01,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107386 s, 381 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:16:06,905::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 18:16:06,907::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:16:06,908::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:16:07,322::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:16:07,328::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:16:07,331::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:16:11,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:16:11,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110608 s, 370 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:16:11,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:16:11,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00668177 s, 613 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:16:21,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:16:21,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165233 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:16:21,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:16:21,820::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00398174 s, 1.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:16:31,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:16:31,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113946 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:16:31,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:16:31,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140249 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:16:41,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:16:41,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.022778 s, 180 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:16:41,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:16:41,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148504 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:16:51,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:16:51,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154787 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:16:52,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:16:52,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799943 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:17:01,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:17:01,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0249338 s, 164 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:17:02,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:17:02,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106245 s, 3.9 MB/s\n'; <rc> = 0
>Thread-14467::WARNING::2013-03-17 18:17:07,364::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:17:07,370::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 18:17:11,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:17:11,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825314 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:17:12,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:17:12,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789477 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:17:17,372::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:17:17,373::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:17:17,768::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:17:17,775::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:17:17,776::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:17:21,469::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 18:17:21,472::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 18:17:21,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:17:21,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000811196 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:17:21,500::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000811196 s, 2.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 18:17:21,501::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 18:17:21,501::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 18:17:21,503::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 18:17:21,504::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:17:21,504::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:17:21,505::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:17:21,505::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:17:21,505::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:17:21,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:17:21,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:17:21,882::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:17:21,883::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 18:17:21,885::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 18:17:21,886::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 18:17:21,887::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 18:17:21,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:17:22,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:17:22,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:17:22,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:17:22,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135832 s, 302 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:17:22,333::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104384 s, 392 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:17:32,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:17:32,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828632 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:17:32,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:17:32,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814514 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:17:42,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:17:42,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079102 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:17:42,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:17:42,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859722 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:17:52,395::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:17:52,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000967412 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:17:52,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:17:52,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793429 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:18:02,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:18:02,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148121 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:18:02,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:18:02,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00698631 s, 586 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:18:12,471::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:18:12,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0212782 s, 192 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:18:12,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:18:12,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00278021 s, 1.5 MB/s\n'; <rc> = 0
>Thread-14468::WARNING::2013-03-17 18:18:17,803::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:18:17,806::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 18:18:22,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:18:22,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884818 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:18:22,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:18:22,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00743398 s, 551 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:18:27,808::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:18:27,809::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:18:28,299::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:18:28,307::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:18:28,309::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:18:32,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:18:32,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075665 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:18:32,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:18:32,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0201257 s, 204 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:18:42,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:18:42,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146376 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:18:42,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:18:42,973::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00712009 s, 575 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:18:52,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:18:52,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823783 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:18:53,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:18:53,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014892 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:19:02,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:19:02,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084242 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:19:03,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:19:03,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111202 s, 368 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:19:12,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:19:12,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143109 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:19:13,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:19:13,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00230957 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:19:22,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:19:22,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129633 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:19:23,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:19:23,372::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105737 s, 3.9 MB/s\n'; <rc> = 0
>Thread-14469::WARNING::2013-03-17 18:19:28,334::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:19:28,339::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 18:19:32,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:19:32,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857448 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:19:33,419::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 18:19:33,421::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 18:19:33,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:19:33,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00169421 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:19:33,497::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00169421 s, 1.2 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 18:19:33,497::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 18:19:33,497::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 18:19:33,499::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 18:19:33,500::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:19:33,501::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:19:33,501::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:19:33,502::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:19:33,502::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:19:33,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:19:33,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:19:33,931::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:19:33,932::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 18:19:33,935::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 18:19:33,936::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 18:19:33,936::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 18:19:33,938::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:19:34,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:19:34,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:19:34,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173039 s, 2.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:19:38,341::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:19:38,342::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:19:38,765::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:19:38,771::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:19:38,773::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:19:42,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:19:42,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114311 s, 358 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:19:44,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:19:44,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821942 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:19:52,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:19:52,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0220715 s, 186 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:19:54,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:19:54,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145513 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:20:02,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:20:03,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0209198 s, 196 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:20:04,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:20:04,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787992 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:20:13,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:20:13,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814094 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:20:14,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:20:14,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014464 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:20:23,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:20:23,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0221074 s, 185 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:20:24,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:20:24,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854378 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:20:33,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:20:33,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000714729 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:20:34,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:20:34,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794662 s, 5.2 MB/s\n'; <rc> = 0
>Thread-14470::WARNING::2013-03-17 18:20:38,804::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:20:38,807::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 18:20:43,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:20:43,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756831 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:20:44,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:20:44,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00279714 s, 1.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:20:48,810::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:20:48,811::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:20:49,278::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:20:49,285::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:20:49,286::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:20:53,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:20:53,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173802 s, 236 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:20:54,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:20:55,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148711 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:21:03,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:21:03,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799693 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:21:05,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:21:05,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000937297 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:21:13,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:21:13,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855962 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:21:15,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:21:15,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788209 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:21:23,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:21:23,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919328 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:21:25,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:21:25,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015178 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:21:33,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:21:33,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00324404 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:21:35,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:21:35,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00097349 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:21:43,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:21:43,456::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779326 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:21:45,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:21:45,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145129 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14471::WARNING::2013-03-17 18:21:49,319::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:21:49,324::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 18:21:53,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:21:53,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00390587 s, 1.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:21:55,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:21:55,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847015 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:21:59,327::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 18:21:59,328::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:21:59,329::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:21:59,745::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:21:59,752::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:21:59,753::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:22:03,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:22:03,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119824 s, 342 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:22:05,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:22:05,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149932 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:22:13,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:22:13,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140543 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:22:15,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:22:15,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0019494 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:22:23,598::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 18:22:23,599::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 18:22:23,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:22:23,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000747612 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:22:23,631::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000747612 s, 2.7 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 18:22:23,632::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 18:22:23,632::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 18:22:23,634::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 18:22:23,635::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:22:23,636::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:22:23,636::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:22:23,636::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:22:23,637::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:22:23,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:22:24,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:22:24,051::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:22:24,052::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 18:22:24,055::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 18:22:24,058::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 18:22:24,058::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 18:22:24,060::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:22:24,404::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:22:24,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:22:24,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00586101 s, 699 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:22:25,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:22:25,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810584 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:22:34,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:22:34,478::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799342 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:22:35,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:22:35,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145856 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:22:44,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:22:44,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839004 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:22:45,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:22:45,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151516 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:22:54,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:22:54,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101637 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:22:55,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:22:56,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167511 s, 2.4 MB/s\n'; <rc> = 0
>Thread-14472::WARNING::2013-03-17 18:22:59,791::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:22:59,794::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 18:23:04,560::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:23:04,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829941 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:23:06,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:23:06,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825824 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:23:09,799::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:23:09,800::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:23:10,223::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:23:10,230::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:23:10,231::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:23:14,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:23:14,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857679 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:23:16,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:23:16,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840426 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:23:24,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:23:24,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888133 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:23:26,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:23:26,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802199 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:23:34,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:23:34,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845356 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:23:36,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:23:36,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137458 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:23:44,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:23:44,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743244 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:23:46,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:23:46,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757012 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:23:54,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:23:54,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128925 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:23:56,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:23:56,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000967045 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:24:04,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:24:04,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130498 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:24:06,522::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:24:06,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00232765 s, 1.8 MB/s\n'; <rc> = 0
>Thread-14473::WARNING::2013-03-17 18:24:10,260::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:24:10,265::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 18:24:14,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:24:14,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.023528 s, 174 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:24:16,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:24:16,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783059 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:24:20,269::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:24:20,270::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:24:20,701::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:24:20,708::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:24:20,709::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:24:24,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:24:24,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901118 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:24:26,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:24:26,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130694 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:24:34,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:24:34,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083147 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:24:36,757::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 18:24:36,759::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 18:24:36,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:24:36,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000756357 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:24:36,796::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000756357 s, 2.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 18:24:36,796::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 18:24:36,797::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 18:24:36,798::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 18:24:36,800::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:24:36,801::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:24:36,801::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:24:36,801::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:24:36,802::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:24:36,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:24:37,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:24:37,220::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:24:37,221::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 18:24:37,223::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 18:24:37,225::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 18:24:37,225::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 18:24:37,227::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:24:37,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:24:37,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:24:37,622::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121073 s, 338 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:24:44,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:24:45,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882116 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:24:47,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:24:47,698::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159656 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:24:55,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:24:55,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156041 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:24:57,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:24:57,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824586 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:25:05,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:25:05,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146248 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:25:07,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:25:07,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138258 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:25:15,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:25:15,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127971 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:25:17,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:25:17,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00199261 s, 2.1 MB/s\n'; <rc> = 0
>Thread-14474::WARNING::2013-03-17 18:25:20,737::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:25:20,740::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 18:25:25,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:25:25,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925489 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:25:27,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:25:28,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789244 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:25:30,751::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:25:30,752::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:25:31,232::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:25:31,239::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:25:31,240::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:25:35,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:25:35,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184165 s, 222 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:25:38,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:25:38,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865952 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:25:45,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:25:45,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758993 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:25:48,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:25:48,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762454 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:25:55,281::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:25:55,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102566 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:25:58,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:25:58,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000990605 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:26:05,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:26:05,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149896 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:26:08,254::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:26:08,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828865 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:26:15,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:26:15,409::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165261 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:26:18,324::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:26:18,366::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144105 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:26:25,414::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:26:25,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152017 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:26:28,405::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:26:28,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774218 s, 5.3 MB/s\n'; <rc> = 0
>Thread-14475::WARNING::2013-03-17 18:26:31,266::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:26:31,270::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 18:26:35,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:26:35,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081919 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:26:38,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:26:38,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104011 s, 3.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:26:41,272::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:26:41,273::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:26:41,730::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:26:41,738::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:26:41,740::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:26:45,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:26:45,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831519 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:26:48,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:26:48,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127207 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:26:55,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:26:55,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140118 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:26:58,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:26:58,692::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837423 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:27:05,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:27:05,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161594 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:27:08,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:27:08,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142126 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:27:15,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:27:15,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0229759 s, 178 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:27:18,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:27:18,881::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0210296 s, 195 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:27:25,710::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 18:27:25,711::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 18:27:25,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:27:25,739::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000826097 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:27:25,740::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000826097 s, 2.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 18:27:25,741::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 18:27:25,741::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 18:27:25,743::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 18:27:25,744::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:27:25,745::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:27:25,745::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:27:25,746::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:27:25,746::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:27:25,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:27:26,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:27:26,178::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:27:26,179::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 18:27:26,181::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 18:27:26,182::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 18:27:26,183::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 18:27:26,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:27:26,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:27:26,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:27:26,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00323605 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:27:28,922::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:27:28,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172211 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:27:36,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:27:36,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111598 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:27:39,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:27:39,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0236553 s, 173 kB/s\n'; <rc> = 0
>Thread-14476::WARNING::2013-03-17 18:27:41,774::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:27:41,777::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 18:27:46,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:27:46,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0229257 s, 179 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:27:49,129::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:27:49,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804408 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:27:51,778::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 18:27:51,780::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:27:51,780::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:27:52,203::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:27:52,210::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:27:52,211::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:27:56,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:27:56,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0232347 s, 176 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:27:59,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:27:59,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847858 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:28:06,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:28:06,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143133 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:28:09,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:28:09,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142395 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:28:16,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:28:16,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144025 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:28:19,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:28:19,424::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753335 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:28:26,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:28:26,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000997065 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:28:29,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:28:29,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000958577 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:28:36,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:28:36,880::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752502 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:28:39,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:28:39,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162889 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:28:46,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:28:46,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162649 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:28:49,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:28:49,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00182025 s, 2.3 MB/s\n'; <rc> = 0
>Thread-14477::WARNING::2013-03-17 18:28:52,242::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:28:52,246::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 18:28:56,938::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:28:56,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008077 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:28:59,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:28:59,738::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818547 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:29:02,248::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:29:02,249::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:29:02,709::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:29:02,716::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:29:02,718::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:29:06,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:29:06,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853157 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:29:09,778::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:29:09,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894499 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:29:17,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:29:17,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143538 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:29:19,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:29:19,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856654 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:29:27,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:29:27,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0212471 s, 193 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:29:29,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:29:29,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000947815 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:29:37,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:29:37,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014584 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:29:40,015::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 18:29:40,016::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 18:29:40,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:29:40,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000808071 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:29:40,047::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000808071 s, 2.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 18:29:40,047::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 18:29:40,047::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 18:29:40,049::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 18:29:40,050::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:29:40,051::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:29:40,051::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:29:40,052::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:29:40,052::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:29:40,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:29:40,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:29:40,479::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:29:40,480::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 18:29:40,483::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 18:29:40,484::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 18:29:40,484::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 18:29:40,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:29:40,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:29:40,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:29:40,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00996681 s, 411 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:29:47,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:29:47,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161057 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:29:50,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:29:50,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149742 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:29:57,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:29:57,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000941188 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:30:01,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:30:01,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114535 s, 3.6 MB/s\n'; <rc> = 0
>Thread-14478::WARNING::2013-03-17 18:30:02,747::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:30:02,752::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 18:30:07,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:30:07,270::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128964 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:30:11,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:30:11,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144454 s, 2.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:30:12,819::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:30:12,821::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:30:13,299::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:30:13,306::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:30:13,308::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:30:17,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:30:17,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00269339 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:30:21,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:30:21,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840417 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:30:27,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:30:27,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136735 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:30:31,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:30:31,309::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821391 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:30:37,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:30:37,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785476 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:30:41,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:30:41,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143111 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:30:47,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:30:47,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189161 s, 217 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:30:51,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:30:51,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0305465 s, 134 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:30:57,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:30:57,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111156 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:31:01,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:31:01,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824583 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:31:07,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:31:07,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827714 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:31:11,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:31:11,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100954 s, 4.1 MB/s\n'; <rc> = 0
>Thread-14479::WARNING::2013-03-17 18:31:13,334::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:31:13,341::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 18:31:17,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:31:17,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795606 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:31:21,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:31:21,777::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000933134 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:31:23,344::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:31:23,345::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:31:23,954::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:31:24,018::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:31:24,068::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:31:27,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:31:27,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0213948 s, 191 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:31:31,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:31:31,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848184 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:31:37,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:31:37,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194599 s, 210 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:31:41,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:31:41,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842796 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:31:47,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:31:47,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000715446 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:31:51,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:31:52,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008227 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:31:57,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:31:57,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101974 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:32:02,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:32:02,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165195 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:32:07,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:32:07,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850506 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:32:12,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:32:12,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011123 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:32:17,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:32:17,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118938 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:32:22,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:32:22,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017326 s, 236 kB/s\n'; <rc> = 0
>Thread-14480::WARNING::2013-03-17 18:32:24,123::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:32:24,128::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 18:32:27,842::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 18:32:27,844::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 18:32:27,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:32:27,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00106231 s, 1.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:32:27,884::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00106231 s, 1.9 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 18:32:27,885::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 18:32:27,885::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 18:32:27,886::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 18:32:27,887::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:32:27,888::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:32:27,888::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:32:27,889::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:32:27,889::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:32:27,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:32:28,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:32:28,399::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:32:28,399::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 18:32:28,402::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 18:32:28,403::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 18:32:28,403::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 18:32:28,405::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:32:28,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:32:28,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:32:28,819::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00625484 s, 655 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:32:32,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:32:32,607::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000717012 s, 5.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:32:34,130::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:32:34,131::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:32:34,546::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:32:34,553::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:32:34,555::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:32:38,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:32:38,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00200258 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:32:42,647::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:32:42,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101747 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:32:48,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:32:48,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137569 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:32:52,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:32:52,744::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852569 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:32:58,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:32:58,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122443 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:33:02,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:33:02,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000706443 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:33:08,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:33:08,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925271 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:33:12,854::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:33:12,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151478 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:33:18,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:33:19,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769438 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:33:22,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:33:22,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149736 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:33:29,034::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:33:29,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150892 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:33:33,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:33:33,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751045 s, 5.5 MB/s\n'; <rc> = 0
>Thread-14481::WARNING::2013-03-17 18:33:34,589::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:33:34,592::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 18:33:39,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:33:39,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806718 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:33:43,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:33:43,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896385 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:33:44,597::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 18:33:44,599::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:33:44,600::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:33:45,009::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:33:45,016::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:33:45,018::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:33:49,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:33:49,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787908 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:33:53,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:33:53,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786611 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:33:59,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:33:59,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193311 s, 212 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:34:03,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:34:03,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00180476 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:34:09,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:34:09,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013751 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:34:13,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:34:13,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780203 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:34:19,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:34:19,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796257 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:34:23,685::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:34:23,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164155 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:34:29,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:34:29,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0214342 s, 191 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:34:33,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:34:33,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859557 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:34:39,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:34:39,408::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00281933 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:34:43,839::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 18:34:43,847::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 18:34:43,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:34:43,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0178014 s, 115 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:34:43,953::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0178014 s, 115 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 18:34:43,954::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 18:34:43,954::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 18:34:43,956::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 18:34:43,958::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:34:43,959::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:34:43,959::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:34:43,960::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:34:43,962::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:34:43,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:34:44,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:34:44,589::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:34:44,594::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 18:34:44,608::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 18:34:44,611::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 18:34:44,612::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 18:34:44,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:34:44,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:34:45,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:34:45,060::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00216448 s, 1.9 MB/s\n'; <rc> = 0
>Thread-14482::WARNING::2013-03-17 18:34:45,070::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:34:45,090::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 18:34:49,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:34:49,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833484 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:34:55,105::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:34:55,107::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:34:55,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:34:55,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141227 s, 2.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:34:55,576::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:34:55,582::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:34:55,583::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:34:59,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:34:59,494::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127571 s, 321 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:35:05,227::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:35:05,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149946 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:35:09,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:35:09,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810164 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:35:15,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:35:15,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148379 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:35:19,530::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:35:19,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855428 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:35:25,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:35:25,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150525 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:35:29,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:35:29,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131338 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:35:35,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:35:35,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0254855 s, 161 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:35:39,605::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:35:39,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164545 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:35:45,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:35:45,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172254 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:35:49,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:35:49,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0023111 s, 1.8 MB/s\n'; <rc> = 0
>Thread-14483::WARNING::2013-03-17 18:35:55,607::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:35:55,610::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:35:55,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:35:55,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766628 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:35:59,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:35:59,733::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915316 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:36:05,613::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:36:05,614::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:36:05,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:36:05,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159177 s, 2.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:36:06,100::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:36:06,107::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:36:06,108::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:36:09,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:36:09,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144897 s, 283 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:36:15,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:36:15,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000742231 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:36:19,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:36:19,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788479 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:36:25,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:36:25,928::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000726021 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:36:29,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:36:29,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142603 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:36:36,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:36:36,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777439 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:36:39,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:36:39,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870694 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:36:46,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:36:46,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143697 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:36:49,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:36:49,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0398605 s, 103 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:36:56,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:36:56,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144595 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:36:59,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:37:00,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133773 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14484::WARNING::2013-03-17 18:37:06,143::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:37:06,147::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:37:06,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:37:06,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088568 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:37:10,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:37:10,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894616 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:37:16,157::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:37:16,159::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:37:16,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:37:16,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156164 s, 2.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:37:16,607::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:37:16,613::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:37:16,614::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:37:20,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:37:20,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084242 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:37:26,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:37:26,511::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140159 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:37:30,110::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 18:37:30,111::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 18:37:30,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:37:30,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000830389 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:37:30,138::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000830389 s, 2.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 18:37:30,138::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 18:37:30,139::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 18:37:30,140::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 18:37:30,141::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:37:30,142::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:37:30,142::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:37:30,143::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:37:30,143::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:37:30,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:37:30,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:37:30,549::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:37:30,550::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 18:37:30,552::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 18:37:30,554::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 18:37:30,554::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 18:37:30,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:37:30,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:37:30,910::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:37:30,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169534 s, 242 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:37:36,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:37:36,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018924 s, 216 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:37:40,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:37:41,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799577 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:37:46,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:37:46,671::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139908 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:37:51,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:37:51,040::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143567 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:37:56,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:37:56,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087929 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:38:01,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:38:01,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144238 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:38:06,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:38:06,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182845 s, 224 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:38:11,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:38:11,130::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161584 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14485::WARNING::2013-03-17 18:38:16,643::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:38:16,649::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:38:16,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:38:16,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100496 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:38:21,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:38:21,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160113 s, 2.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:38:26,652::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:38:26,654::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:38:26,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:38:26,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173806 s, 2.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:38:27,133::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:38:27,140::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:38:27,145::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:38:31,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:38:31,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885269 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:38:37,034::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:38:37,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807775 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:38:41,209::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:38:41,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140591 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:38:47,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:38:47,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103652 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:38:51,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:38:51,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0198735 s, 206 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:38:57,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:38:57,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800627 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:39:01,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:39:01,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0223657 s, 183 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:39:07,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:39:07,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147127 s, 278 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:39:11,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:39:11,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0218559 s, 187 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:39:17,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:39:17,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142419 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:39:21,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:39:21,481::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142564 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14486::WARNING::2013-03-17 18:39:27,177::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:39:27,181::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:39:27,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:39:27,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882694 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:39:31,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:39:31,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0213163 s, 192 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:39:37,186::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 18:39:37,187::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:39:37,188::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:39:37,522::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:39:37,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139269 s, 2.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:39:37,648::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:39:37,654::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:39:37,655::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:39:41,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:39:41,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0214244 s, 191 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:39:47,616::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 18:39:47,617::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 18:39:47,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:39:47,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0015839 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:39:47,664::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0015839 s, 1.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 18:39:47,665::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 18:39:47,665::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 18:39:47,667::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 18:39:47,668::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:39:47,668::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:39:47,669::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:39:47,669::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:39:47,670::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:39:47,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:39:48,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:39:48,109::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:39:48,110::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 18:39:48,113::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 18:39:48,114::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 18:39:48,114::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 18:39:48,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:39:48,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:39:48,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:39:48,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00695315 s, 589 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:39:51,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:39:51,667::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123514 s, 332 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:39:58,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:39:58,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778433 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:40:01,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:40:01,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192639 s, 213 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:40:08,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:40:08,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147696 s, 277 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:40:11,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:40:11,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078369 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:40:18,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:40:18,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822251 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:40:21,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:40:21,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0208454 s, 196 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:40:28,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:40:28,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848713 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:40:31,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:40:32,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863491 s, 4.7 MB/s\n'; <rc> = 0
>Thread-14487::WARNING::2013-03-17 18:40:37,680::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:40:37,687::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:40:38,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:40:38,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924888 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:40:42,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:40:42,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814081 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:40:47,691::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:40:47,692::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:40:48,108::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:40:48,114::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:40:48,116::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:40:48,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:40:48,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0091686 s, 447 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:40:52,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:40:52,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.020867 s, 196 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:40:59,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:40:59,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153986 s, 266 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:41:02,108::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:41:02,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183479 s, 223 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:41:09,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:41:09,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175071 s, 234 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:41:12,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:41:12,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140862 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:41:19,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:41:19,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160697 s, 255 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:41:22,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:41:22,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144854 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:41:29,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:41:29,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158098 s, 259 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:41:32,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:41:32,309::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083599 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:41:39,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:41:39,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158265 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:41:42,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:41:42,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00197189 s, 2.1 MB/s\n'; <rc> = 0
>Thread-14488::WARNING::2013-03-17 18:41:48,142::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:41:48,147::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:41:49,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:41:49,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151283 s, 271 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:41:52,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:41:52,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008179 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:41:58,149::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:41:58,150::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:41:58,558::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:41:58,565::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:41:58,566::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:41:59,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:41:59,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017253 s, 237 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:42:02,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:42:02,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101491 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:42:09,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:42:09,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147822 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:42:12,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:42:12,490::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150835 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:42:19,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:42:19,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110714 s, 370 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:42:22,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:42:22,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179217 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:42:29,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:42:29,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015666 s, 261 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:42:32,543::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 18:42:32,544::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 18:42:32,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:42:32,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00088916 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:42:32,580::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00088916 s, 2.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 18:42:32,581::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 18:42:32,581::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 18:42:32,583::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 18:42:32,585::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:42:32,586::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:42:32,586::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:42:32,587::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:42:32,587::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:42:32,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:42:33,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:42:33,037::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:42:33,038::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 18:42:33,041::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 18:42:33,042::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 18:42:33,043::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 18:42:33,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:42:33,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:42:33,396::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:42:33,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100726 s, 407 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:42:39,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:42:39,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163813 s, 250 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:42:43,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:42:43,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781117 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:42:50,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:42:50,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0205153 s, 200 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:42:53,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:42:53,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837223 s, 4.9 MB/s\n'; <rc> = 0
>Thread-14489::WARNING::2013-03-17 18:42:58,599::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:42:58,605::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:43:00,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:43:00,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800042 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:43:03,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:43:03,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852097 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:43:08,608::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:43:08,609::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:43:09,020::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:43:09,027::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:43:09,028::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:43:10,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:43:10,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00277345 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:43:13,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:43:13,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156725 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:43:20,260::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:43:20,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814848 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:43:23,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:43:23,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751947 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:43:30,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:43:30,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795626 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:43:33,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:43:33,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171664 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:43:40,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:43:40,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153519 s, 267 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:43:43,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:43:43,709::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805139 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:43:50,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:43:50,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185462 s, 221 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:43:53,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:43:53,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143391 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:44:00,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:44:00,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124968 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:44:03,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:44:03,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753358 s, 5.4 MB/s\n'; <rc> = 0
>Thread-14490::WARNING::2013-03-17 18:44:09,055::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:44:09,060::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:44:10,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:44:10,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906449 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:44:13,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:44:13,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911796 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:44:19,064::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:44:19,065::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:44:19,537::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:44:19,544::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:44:19,546::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:44:20,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:44:20,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00301663 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:44:23,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:44:23,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875969 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:44:30,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:44:30,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000945577 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:44:33,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:44:33,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0211489 s, 194 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:44:40,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:44:40,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835374 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:44:43,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:44:43,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0207824 s, 197 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:44:50,981::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 18:44:50,982::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 18:44:50,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:44:51,046::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0169367 s, 121 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:44:51,048::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0169367 s, 121 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 18:44:51,048::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 18:44:51,048::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 18:44:51,050::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 18:44:51,051::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:44:51,051::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:44:51,052::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:44:51,052::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:44:51,053::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:44:51,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:44:51,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:44:51,450::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:44:51,451::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 18:44:51,454::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 18:44:51,455::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 18:44:51,455::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 18:44:51,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:44:51,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:44:51,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:44:51,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0096672 s, 424 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:44:54,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:44:54,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0196143 s, 209 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:45:01,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:45:01,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144954 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:45:04,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:45:04,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0955131 s, 42.9 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:45:11,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:45:12,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189377 s, 216 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:45:14,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:45:14,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00282248 s, 1.5 MB/s\n'; <rc> = 0
>Thread-14491::WARNING::2013-03-17 18:45:19,584::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:45:19,588::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:45:22,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:45:22,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842037 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:45:24,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:45:24,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155054 s, 2.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:45:29,589::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 18:45:29,591::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:45:29,592::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:45:30,048::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:45:30,055::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:45:30,056::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:45:32,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:45:32,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798931 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:45:34,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:45:34,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916879 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:45:42,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:45:42,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150032 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:45:44,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:45:44,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147278 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:45:52,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:45:52,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787256 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:45:54,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:45:54,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157413 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:46:02,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:46:02,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000742002 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:46:04,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:46:04,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0263318 s, 156 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:46:12,429::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:46:12,484::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0238362 s, 172 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:46:14,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:46:14,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000937164 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:46:22,530::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:46:22,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191868 s, 213 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:46:24,528::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:46:24,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138215 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14492::WARNING::2013-03-17 18:46:30,088::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:46:30,092::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:46:32,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:46:32,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173832 s, 236 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:46:34,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:46:34,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827647 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:46:40,094::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:46:40,096::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:46:40,484::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:46:40,491::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:46:40,493::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:46:42,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:46:42,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154901 s, 264 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:46:44,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:46:44,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810497 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:46:52,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:46:52,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169968 s, 241 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:46:54,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:46:54,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154007 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:47:02,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:47:02,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00197751 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:47:04,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:47:04,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0214547 s, 191 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:47:12,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:47:13,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0157682 s, 260 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:47:14,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:47:14,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795985 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:47:23,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:47:23,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139647 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:47:24,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:47:24,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741823 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:47:33,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:47:33,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841754 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:47:34,835::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 18:47:34,839::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 18:47:34,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:47:34,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00307559 s, 666 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:47:34,872::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00307559 s, 666 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 18:47:34,873::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 18:47:34,873::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 18:47:34,875::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 18:47:34,876::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:47:34,877::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:47:34,877::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:47:34,877::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:47:34,878::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:47:34,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:47:35,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:47:35,329::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:47:35,330::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 18:47:35,333::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 18:47:35,334::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 18:47:35,334::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 18:47:35,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:47:35,700::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:47:35,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:47:35,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193466 s, 212 kB/s\n'; <rc> = 0
>Thread-14493::WARNING::2013-03-17 18:47:40,521::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:47:40,525::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:47:43,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:47:43,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168389 s, 243 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:47:45,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:47:45,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00221158 s, 1.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:47:50,527::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:47:50,528::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:47:50,949::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:47:50,955::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:47:50,957::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:47:53,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:47:53,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016436 s, 249 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:47:55,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:47:55,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00379909 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:48:03,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:48:03,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171765 s, 238 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:48:05,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:48:05,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145953 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:48:13,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:48:13,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782914 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:48:15,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:48:15,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138224 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:48:23,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:48:23,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167682 s, 244 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:48:25,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:48:25,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792572 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:48:33,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:48:33,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122448 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:48:35,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:48:36,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911707 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:48:43,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:48:43,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206089 s, 199 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:48:46,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:48:46,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175977 s, 2.3 MB/s\n'; <rc> = 0
>Thread-14494::WARNING::2013-03-17 18:48:50,984::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:48:50,991::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:48:53,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:48:53,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782222 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:48:56,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:48:56,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00502656 s, 815 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:49:00,994::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:49:00,995::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:49:01,460::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:49:01,467::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:49:01,468::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:49:03,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:49:03,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178367 s, 230 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:49:06,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:49:06,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000991551 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:49:14,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:49:14,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00252862 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:49:16,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:49:16,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902527 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:49:24,079::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:49:24,140::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0284338 s, 144 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:49:26,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:49:26,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783513 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:49:34,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:49:34,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107321 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:49:36,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:49:36,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100342 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:49:44,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:49:44,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774949 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:49:46,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:49:46,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014053 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:49:54,323::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 18:49:54,324::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 18:49:54,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:49:54,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0166053 s, 123 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:49:54,378::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0166053 s, 123 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 18:49:54,379::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 18:49:54,379::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 18:49:54,381::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 18:49:54,384::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:49:54,385::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:49:54,385::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:49:54,385::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:49:54,386::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:49:54,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:49:54,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:49:54,815::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:49:54,816::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 18:49:54,819::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 18:49:54,820::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 18:49:54,820::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 18:49:54,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:49:55,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:49:55,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:49:55,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00909608 s, 450 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:49:56,305::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:49:56,364::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176039 s, 233 kB/s\n'; <rc> = 0
>Thread-14495::WARNING::2013-03-17 18:50:01,500::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:50:01,505::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:50:05,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:50:05,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091137 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:50:06,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:50:06,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149492 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:50:11,508::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:50:11,509::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:50:11,936::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:50:11,943::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:50:11,944::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:50:15,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:50:15,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761734 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:50:16,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:50:16,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077337 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:50:25,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:50:25,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826626 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:50:26,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:50:26,478::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853104 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:50:35,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:50:35,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141394 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:50:36,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:50:36,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00265667 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:50:45,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:50:45,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146096 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:50:46,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:50:46,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796874 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:50:55,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:50:55,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117275 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:50:56,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:50:56,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108799 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:51:05,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:51:05,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000689767 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:51:06,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:51:06,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144374 s, 284 kB/s\n'; <rc> = 0
>Thread-14496::WARNING::2013-03-17 18:51:11,971::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:51:11,977::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:51:15,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:51:15,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160404 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:51:16,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:51:16,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893383 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:51:21,979::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 18:51:21,980::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:51:21,982::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:51:22,413::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:51:22,419::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:51:22,420::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:51:25,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:51:25,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080206 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:51:26,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:51:26,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911178 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:51:35,993::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:51:36,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012034 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:51:36,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:51:36,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924794 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:51:46,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:51:46,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0311805 s, 131 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:51:46,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:51:46,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194832 s, 210 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:51:56,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:51:56,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834007 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:51:56,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:51:56,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147706 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:52:06,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:52:06,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147285 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:52:06,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:52:06,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129075 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:52:16,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:52:16,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777653 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:52:16,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:52:16,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204264 s, 201 kB/s\n'; <rc> = 0
>Thread-14497::WARNING::2013-03-17 18:52:22,464::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:52:22,468::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:52:26,415::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:52:26,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00090684 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:52:26,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:52:27,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151635 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:52:32,470::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:52:32,471::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:52:32,891::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:52:32,898::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:52:32,900::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:52:36,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:52:36,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782077 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:52:37,016::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 18:52:37,018::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 18:52:37,019::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:52:37,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0200928 s, 102 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:52:37,084::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0200928 s, 102 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 18:52:37,084::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 18:52:37,085::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 18:52:37,086::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 18:52:37,087::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:52:37,088::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:52:37,088::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:52:37,089::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:52:37,089::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:52:37,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:52:37,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:52:37,472::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:52:37,473::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 18:52:37,476::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 18:52:37,477::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 18:52:37,478::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 18:52:37,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:52:37,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:52:37,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:52:37,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.021906 s, 187 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:52:46,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:52:46,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00216925 s, 1.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:52:47,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:52:47,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085949 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:52:56,617::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:52:56,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161152 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:52:57,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:52:57,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744845 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:53:06,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:53:06,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786582 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:53:07,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:53:08,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000926769 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:53:16,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:53:16,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0230108 s, 178 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:53:18,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:53:18,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0200357 s, 204 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:53:26,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:53:26,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132157 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:53:28,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:53:28,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0214564 s, 191 kB/s\n'; <rc> = 0
>Thread-14498::WARNING::2013-03-17 18:53:32,926::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:53:32,930::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:53:36,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:53:36,992::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0198852 s, 206 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:53:38,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:53:38,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0319263 s, 128 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:53:42,932::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:53:42,933::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:53:43,339::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:53:43,346::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:53:43,347::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:53:47,033::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:53:47,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00447846 s, 915 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:53:48,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:53:48,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862083 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:53:57,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:53:57,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179313 s, 228 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:53:58,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:53:58,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164892 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:54:07,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:54:07,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0243263 s, 168 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:54:08,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:54:08,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878978 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:54:17,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:54:17,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765898 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:54:18,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:54:18,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797692 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:54:27,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:54:27,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150729 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:54:28,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:54:28,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013845 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:54:37,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:54:37,484::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00294319 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:54:38,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:54:38,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140548 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14499::WARNING::2013-03-17 18:54:43,377::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:54:43,381::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:54:47,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:54:47,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148782 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:54:48,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:54:48,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870632 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:54:53,384::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:54:53,385::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:54:53,798::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:54:53,805::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:54:53,806::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:54:57,619::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 18:54:57,622::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 18:54:57,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:54:57,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000784537 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:54:57,649::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000784537 s, 2.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 18:54:57,649::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 18:54:57,650::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 18:54:57,652::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 18:54:57,653::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:54:57,654::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:54:57,654::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:54:57,654::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:54:57,655::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 18:54:57,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:54:58,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:54:58,092::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:54:58,093::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 18:54:58,095::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 18:54:58,097::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 18:54:58,097::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 18:54:58,099::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:54:58,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:54:58,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:54:58,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:54:58,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00528508 s, 775 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:54:58,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170416 s, 240 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:55:08,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:55:08,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:55:08,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013131 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:55:08,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807756 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:55:18,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:55:18,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:55:18,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00234222 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:55:18,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107767 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:55:28,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:55:28,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861429 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:55:28,871::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:55:28,912::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143222 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:55:38,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:55:38,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143424 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:55:38,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:55:38,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013755 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:55:48,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:55:48,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145296 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:55:49,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:55:49,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000978939 s, 4.2 MB/s\n'; <rc> = 0
>Thread-14500::WARNING::2013-03-17 18:55:53,837::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:55:53,842::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 18:55:58,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:55:59,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895788 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:55:59,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:55:59,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819283 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:56:03,845::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:56:03,846::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:56:04,285::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:56:04,292::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:56:04,293::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:56:09,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:56:09,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187027 s, 219 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:56:09,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:56:09,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172282 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:56:19,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:56:19,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0225962 s, 181 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:56:19,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:56:19,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836783 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:56:29,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:56:29,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0325891 s, 126 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:56:29,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:56:29,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759886 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:56:39,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:56:39,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0229052 s, 179 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:56:39,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:56:39,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116125 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:56:49,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:56:49,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0219544 s, 187 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:56:49,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:56:49,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016044 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:56:59,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:56:59,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.023867 s, 172 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:56:59,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:56:59,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162079 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14501::WARNING::2013-03-17 18:57:04,318::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:57:04,322::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 18:57:09,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:57:09,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0213393 s, 192 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:57:09,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:57:09,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00208274 s, 2.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:57:14,323::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 18:57:14,324::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:57:14,325::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:57:14,769::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:57:14,776::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:57:14,777::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:57:19,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:57:19,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828116 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:57:19,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:57:19,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839529 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:57:29,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:57:29,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844188 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:57:29,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:57:29,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773392 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:57:39,547::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 18:57:39,548::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 18:57:39,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:57:39,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00134376 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:57:39,581::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00134376 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 18:57:39,582::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 18:57:39,582::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 18:57:39,584::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 18:57:39,585::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:57:39,585::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:57:39,586::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:57:39,586::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:57:39,586::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 18:57:39,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:57:39,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:57:39,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160122 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:57:40,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:57:40,028::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 18:57:40,029::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 18:57:40,032::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 18:57:40,033::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 18:57:40,034::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 18:57:40,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:57:40,398::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:57:40,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:57:40,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134654 s, 304 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:57:49,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:57:50,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773738 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:57:50,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:57:50,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751043 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:58:00,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:58:00,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896703 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:58:00,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:58:00,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163334 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:58:10,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:58:10,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168342 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:58:10,530::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:58:10,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120128 s, 3.4 MB/s\n'; <rc> = 0
>Thread-14502::WARNING::2013-03-17 18:58:14,806::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:58:14,812::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:58:20,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:58:20,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0322821 s, 127 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:58:20,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:58:20,598::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114053 s, 3.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:58:24,816::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:58:24,817::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:58:25,224::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:58:25,231::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:58:25,232::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:58:30,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:58:30,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850888 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:58:30,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:58:30,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765484 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:58:40,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:58:40,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774807 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:58:40,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:58:40,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141942 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:58:50,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:58:50,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169196 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:58:50,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:58:50,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161887 s, 253 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:59:00,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:59:00,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917697 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:59:00,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:59:00,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838405 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:59:10,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:59:10,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765106 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:59:10,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:59:10,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088707 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:59:20,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:59:20,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789783 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:59:20,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:59:20,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119896 s, 3.4 MB/s\n'; <rc> = 0
>Thread-14503::WARNING::2013-03-17 18:59:25,268::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 18:59:25,275::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 18:59:30,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:59:30,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859457 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:59:30,894::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:59:30,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077214 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 18:59:35,278::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 18:59:35,280::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 18:59:35,699::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 18:59:35,706::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 18:59:35,708::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 18:59:40,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:59:40,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840002 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:59:40,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:59:40,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146972 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 18:59:50,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 18:59:50,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167957 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 18:59:50,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 18:59:51,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819917 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:00:01,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:00:01,045::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 19:00:01,059::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 19:00:01,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:00:01,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124989 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:00:01,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00683052 s, 300 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:00:01,139::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00683052 s, 300 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 19:00:01,139::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 19:00:01,140::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 19:00:01,140::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 19:00:01,141::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:00:01,142::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:00:01,143::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:00:01,144::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:00:01,144::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:00:01,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:00:01,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:00:01,703::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:00:01,704::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 19:00:01,706::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 19:00:01,707::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 19:00:01,708::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 19:00:01,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:00:02,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:00:02,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:00:02,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105297 s, 389 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:00:11,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:00:11,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145332 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:00:12,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:00:12,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149806 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:00:21,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:00:21,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170207 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:00:22,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:00:22,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149874 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:00:31,280::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:00:31,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000997314 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:00:32,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:00:32,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168388 s, 2.4 MB/s\n'; <rc> = 0
>Thread-14504::WARNING::2013-03-17 19:00:35,734::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:00:35,738::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 19:00:41,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:00:41,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0311795 s, 131 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:00:42,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:00:42,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833933 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:00:45,740::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:00:45,741::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:00:46,155::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:00:46,162::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:00:46,163::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:00:51,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:00:51,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920768 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:00:52,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:00:52,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000713403 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:01:01,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:01:01,450::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142487 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:01:02,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:01:02,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162299 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:01:11,455::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:01:11,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00462663 s, 885 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:01:12,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:01:12,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015757 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:01:21,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:01:21,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148818 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:01:22,933::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:01:22,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779922 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:01:31,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:01:31,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899419 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:01:33,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:01:33,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152399 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:01:41,617::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:01:41,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131229 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:01:43,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:01:43,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141365 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14505::WARNING::2013-03-17 19:01:46,192::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:01:46,195::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 19:01:51,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:01:51,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100433 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:01:53,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:01:53,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875846 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:01:56,197::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:01:56,198::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:01:56,622::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:01:56,628::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:01:56,630::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:02:01,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:02:01,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774721 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:02:03,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:02:03,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00992163 s, 413 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:02:11,738::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:02:11,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123177 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:02:13,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:02:13,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00369761 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:02:21,781::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:02:21,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014802 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:02:23,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:02:23,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00194998 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:02:31,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:02:31,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898286 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:02:33,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:02:33,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000980127 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:02:41,866::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 19:02:41,868::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 19:02:41,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:02:41,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0202964 s, 101 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:02:41,927::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0202964 s, 101 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 19:02:41,927::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 19:02:41,927::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 19:02:41,929::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 19:02:41,930::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:02:41,931::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:02:41,931::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:02:41,932::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:02:41,933::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:02:41,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:02:42,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:02:42,381::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:02:42,382::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 19:02:42,384::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 19:02:42,385::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 19:02:42,386::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 19:02:42,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:02:42,801::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:02:42,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:02:42,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116089 s, 353 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:02:43,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:02:43,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135508 s, 302 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:02:52,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:02:52,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000663785 s, 6.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:02:53,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:02:53,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000946557 s, 4.3 MB/s\n'; <rc> = 0
>Thread-14506::WARNING::2013-03-17 19:02:56,663::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:02:56,667::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 19:03:02,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:03:02,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113995 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:03:03,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:03:03,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00825868 s, 496 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:03:06,672::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 19:03:06,674::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:03:06,675::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:03:07,189::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:03:07,196::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:03:07,198::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:03:12,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:03:12,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837705 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:03:13,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:03:13,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104581 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:03:22,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:03:22,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777303 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:03:23,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:03:23,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013001 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:03:32,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:03:33,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820262 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:03:33,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:03:33,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153924 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:03:43,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:03:43,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773284 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:03:44,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:03:44,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165508 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:03:53,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:03:53,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152466 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:03:54,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:03:54,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00191118 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:04:03,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:04:03,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149294 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:04:04,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:04:04,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146493 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14507::WARNING::2013-03-17 19:04:07,233::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:04:07,237::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 19:04:13,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:04:13,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760617 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:04:14,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:04:14,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185538 s, 2.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:04:17,239::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:04:17,240::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:04:17,654::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:04:17,661::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:04:17,662::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:04:23,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:04:23,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139721 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:04:24,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:04:24,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142361 s, 288 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:04:33,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:04:33,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0210811 s, 194 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:04:34,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:04:34,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150088 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:04:43,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:04:43,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00902986 s, 454 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:04:44,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:04:44,607::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820483 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:04:53,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:04:53,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0414716 s, 98.8 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:04:54,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:04:54,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777091 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:05:03,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:05:03,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136169 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:05:04,723::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 19:05:04,725::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 19:05:04,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:05:04,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00152546 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:05:04,765::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00152546 s, 1.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 19:05:04,766::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 19:05:04,766::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 19:05:04,768::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 19:05:04,769::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:05:04,769::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:05:04,770::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:05:04,770::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:05:04,770::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:05:04,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:05:05,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:05:05,171::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:05:05,173::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 19:05:05,175::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 19:05:05,177::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 19:05:05,177::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 19:05:05,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:05:05,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:05:05,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:05:05,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143546 s, 285 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:05:13,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:05:13,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756816 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:05:15,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:05:15,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823866 s, 5.0 MB/s\n'; <rc> = 0
>Thread-14508::WARNING::2013-03-17 19:05:17,688::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:05:17,691::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 19:05:23,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:05:23,738::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862748 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:05:25,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:05:25,744::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0093369 s, 439 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:05:27,693::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:05:27,695::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:05:28,129::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:05:28,136::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:05:28,137::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:05:33,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:05:33,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861539 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:05:35,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:05:35,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811008 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:05:43,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:05:43,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747751 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:05:45,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:05:45,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164295 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:05:53,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:05:53,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134274 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:05:55,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:05:55,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150597 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:06:03,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:06:03,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801623 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:06:06,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:06:06,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148681 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:06:13,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:06:13,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171053 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:06:16,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:06:16,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147543 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:06:23,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:06:23,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770115 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:06:26,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:06:26,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00096785 s, 4.2 MB/s\n'; <rc> = 0
>Thread-14509::WARNING::2013-03-17 19:06:28,164::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:06:28,167::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 19:06:33,993::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:06:34,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822337 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:06:36,280::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:06:36,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793653 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:06:38,170::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:06:38,172::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:06:38,598::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:06:38,608::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:06:38,609::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:06:44,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:06:44,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0386051 s, 106 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:06:46,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:06:46,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153443 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:06:54,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:06:54,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00617399 s, 663 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:06:56,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:06:56,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154276 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:07:04,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:07:04,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00978403 s, 419 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:07:06,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:07:06,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777965 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:07:14,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:07:14,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135466 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:07:16,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:07:16,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105587 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:07:24,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:07:24,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00384299 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:07:26,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:07:26,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728444 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:07:34,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:07:34,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169954 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:07:36,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:07:36,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000934019 s, 4.4 MB/s\n'; <rc> = 0
>Thread-14510::WARNING::2013-03-17 19:07:38,648::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:07:38,652::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 19:07:44,362::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 19:07:44,364::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 19:07:44,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:07:44,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000786895 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:07:44,397::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000786895 s, 2.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 19:07:44,398::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 19:07:44,398::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 19:07:44,399::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 19:07:44,401::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:07:44,401::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:07:44,402::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:07:44,402::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:07:44,403::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:07:44,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:07:44,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:07:44,836::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:07:44,837::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 19:07:44,839::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 19:07:44,840::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 19:07:44,841::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 19:07:44,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:07:45,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:07:45,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:07:45,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013972 s, 293 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:07:46,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:07:46,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768981 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:07:48,654::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:07:48,655::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:07:49,055::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:07:49,061::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:07:49,063::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:07:55,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:07:55,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891196 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:07:56,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:07:56,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135372 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:08:05,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:08:05,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152184 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:08:06,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:08:07,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0311293 s, 132 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:08:15,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:08:15,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807075 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:08:17,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:08:17,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924187 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:08:25,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:08:25,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142016 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:08:27,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:08:27,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761355 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:08:35,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:08:35,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130764 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:08:37,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:08:37,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780383 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:08:45,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:08:45,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00189587 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:08:47,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:08:47,364::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00314786 s, 1.3 MB/s\n'; <rc> = 0
>Thread-14511::WARNING::2013-03-17 19:08:49,090::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:08:49,095::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 19:08:55,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:08:55,526::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809222 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:08:57,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:08:57,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104478 s, 3.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:08:59,097::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 19:08:59,099::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:08:59,100::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:08:59,531::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:08:59,537::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:08:59,539::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:09:05,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:09:05,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821718 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:09:07,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:09:07,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000964759 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:09:15,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:09:15,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079104 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:09:17,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:09:17,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898324 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:09:25,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:09:25,627::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755228 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:09:27,837::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:09:27,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797876 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:09:35,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:09:35,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141653 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:09:37,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:09:37,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130729 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:09:45,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:09:45,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138435 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:09:47,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:09:48,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185856 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:09:55,717::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:09:55,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148274 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:09:58,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:09:58,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105761 s, 3.9 MB/s\n'; <rc> = 0
>Thread-14512::WARNING::2013-03-17 19:09:59,567::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:09:59,570::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 19:10:05,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:10:05,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000951021 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:10:08,178::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 19:10:08,179::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 19:10:08,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:10:08,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00108777 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:10:08,207::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00108777 s, 1.9 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 19:10:08,207::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 19:10:08,208::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 19:10:08,209::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 19:10:08,211::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:10:08,211::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:10:08,212::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:10:08,212::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:10:08,212::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:10:08,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:10:08,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:10:08,644::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:10:08,646::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 19:10:08,648::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 19:10:08,649::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 19:10:08,650::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 19:10:08,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:10:09,010::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:10:09,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:10:09,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106752 s, 384 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:10:09,572::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:10:09,574::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:10:09,993::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:10:10,000::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:10:10,001::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:10:15,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:10:15,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000974035 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:10:19,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:10:19,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154268 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:10:25,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:10:25,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868531 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:10:29,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:10:29,270::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885791 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:10:35,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:10:35,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134538 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:10:39,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:10:39,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737634 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:10:45,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:10:45,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909604 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:10:49,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:10:49,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000968674 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:10:55,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:10:55,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160348 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:10:59,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:10:59,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875128 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:11:06,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:11:06,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124472 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:11:09,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:11:09,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000732119 s, 5.6 MB/s\n'; <rc> = 0
>Thread-14513::WARNING::2013-03-17 19:11:10,028::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:11:10,032::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 19:11:16,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:11:16,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744064 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:11:19,859::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:11:19,899::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771074 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:11:20,035::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:11:20,036::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:11:20,485::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:11:20,492::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:11:20,493::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:11:26,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:11:26,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161247 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:11:29,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:11:29,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900826 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:11:36,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:11:36,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000905087 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:11:40,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:11:40,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167112 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:11:46,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:11:46,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147784 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:11:50,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:11:50,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924872 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:11:56,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:11:56,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016113 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:12:00,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:12:00,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179987 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:12:06,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:12:06,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0567168 s, 72.2 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:12:10,260::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:12:10,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829023 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:12:16,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:12:16,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171226 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:12:20,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:12:20,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00635375 s, 645 kB/s\n'; <rc> = 0
>Thread-14514::WARNING::2013-03-17 19:12:20,521::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:12:20,533::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 19:12:26,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:12:26,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792342 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:12:30,538::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:12:30,539::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:12:30,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:12:30,611::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161336 s, 2.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:12:31,070::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:12:31,075::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:12:31,077::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:12:36,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:12:36,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000995513 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:12:40,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:12:40,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00229929 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:12:46,492::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 19:12:46,493::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 19:12:46,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:12:46,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00190002 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:12:46,534::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00190002 s, 1.1 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 19:12:46,534::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 19:12:46,534::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 19:12:46,536::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 19:12:46,537::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:12:46,537::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:12:46,538::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:12:46,538::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:12:46,538::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:12:46,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:12:46,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:12:46,972::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:12:46,973::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 19:12:46,978::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 19:12:46,979::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 19:12:46,980::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 19:12:46,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:12:47,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:12:47,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:12:47,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132104 s, 310 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:12:50,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:12:50,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144656 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:12:57,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:12:57,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00281844 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:13:00,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:13:00,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175484 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:13:07,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:13:07,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128891 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:13:10,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:13:10,944::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164291 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:13:17,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:13:17,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163486 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:13:20,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:13:21,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167133 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:13:27,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:13:27,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795704 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:13:31,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-14515::WARNING::2013-03-17 19:13:31,110::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:13:31,126::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:13:31,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00240685 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:13:37,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:13:37,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000946693 s, 4.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:13:41,141::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:13:41,142::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:13:41,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:13:41,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00925355 s, 443 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:13:41,655::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:13:41,660::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:13:41,662::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:13:47,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:13:47,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155473 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:13:51,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:13:51,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144697 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:13:57,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:13:57,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862291 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:14:01,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:14:01,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106884 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:14:07,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:14:07,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158766 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:14:11,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:14:11,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896453 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:14:17,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:14:17,742::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000709915 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:14:21,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:14:21,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142636 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:14:27,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:14:27,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144651 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:14:31,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:14:31,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000910278 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:14:37,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:14:37,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149217 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14516::WARNING::2013-03-17 19:14:41,690::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:14:41,696::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:14:41,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:14:41,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817975 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:14:47,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:14:47,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795138 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:14:51,698::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 19:14:51,699::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:14:51,701::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:14:51,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:14:51,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0202193 s, 203 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:14:52,193::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:14:52,199::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:14:52,200::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:14:57,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:14:57,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743356 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:15:01,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:15:01,928::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849105 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:15:07,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:15:07,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119143 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:15:11,967::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 19:15:11,968::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 19:15:11,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:15:11,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000830081 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:15:11,998::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000830081 s, 2.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 19:15:11,998::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 19:15:11,998::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 19:15:12,000::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 19:15:12,001::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:15:12,002::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:15:12,002::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:15:12,003::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:15:12,003::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:15:12,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:15:12,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:15:12,431::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:15:12,433::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 19:15:12,435::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 19:15:12,436::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 19:15:12,437::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 19:15:12,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:15:12,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:15:12,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:15:12,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00968942 s, 423 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:15:17,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:15:18,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0303807 s, 135 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:15:22,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:15:22,950::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147391 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:15:28,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:15:28,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144511 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:15:32,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:15:33,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832162 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:15:38,079::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:15:38,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161473 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:15:43,060::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:15:43,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161696 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:15:48,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:15:48,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00431752 s, 949 kB/s\n'; <rc> = 0
>Thread-14517::WARNING::2013-03-17 19:15:52,225::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:15:52,228::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:15:53,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:15:53,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799025 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:15:58,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:15:58,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765354 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:16:02,231::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:16:02,232::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:16:02,628::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:16:02,634::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:16:02,638::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:16:03,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:16:03,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197102 s, 208 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:16:08,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:16:08,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124846 s, 328 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:16:13,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:16:13,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793845 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:16:18,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:16:18,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904546 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:16:23,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:16:23,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150523 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:16:28,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:16:28,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907002 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:16:33,492::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:16:33,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906235 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:16:38,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:16:38,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779585 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:16:43,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:16:43,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0309103 s, 133 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:16:48,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:16:48,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000905328 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:16:53,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:16:53,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144881 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:16:58,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:16:58,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936908 s, 4.4 MB/s\n'; <rc> = 0
>Thread-14518::WARNING::2013-03-17 19:17:02,668::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:17:02,674::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:17:03,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:17:03,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807175 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:17:08,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:17:08,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141403 s, 2.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:17:12,677::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:17:12,679::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:17:13,100::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:17:13,106::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:17:13,108::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:17:13,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:17:13,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00940203 s, 436 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:17:18,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:17:18,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140297 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:17:24,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:17:24,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016553 s, 247 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:17:28,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:17:28,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149247 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:17:34,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:17:34,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00769249 s, 532 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:17:38,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:17:38,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915746 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:17:44,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:17:44,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132757 s, 309 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:17:48,668::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 19:17:48,669::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 19:17:48,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:17:48,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0140224 s, 146 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:17:48,710::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0140224 s, 146 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 19:17:48,715::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 19:17:48,716::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 19:17:48,718::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 19:17:48,719::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:17:48,719::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:17:48,720::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:17:48,720::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:17:48,721::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:17:48,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:17:49,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:17:49,150::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:17:49,151::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 19:17:49,153::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 19:17:49,155::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 19:17:49,155::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 19:17:49,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:17:49,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:17:49,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:17:49,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798987 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:17:54,305::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:17:54,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145046 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:17:59,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:17:59,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159327 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:18:04,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:18:04,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130145 s, 315 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:18:09,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:18:09,627::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129884 s, 3.2 MB/s\n'; <rc> = 0
>Thread-14519::WARNING::2013-03-17 19:18:13,135::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:18:13,138::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:18:14,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:18:14,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936658 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:18:19,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:18:19,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113371 s, 361 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:18:23,141::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:18:23,142::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:18:23,586::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:18:23,593::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:18:23,595::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:18:24,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:18:24,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169796 s, 241 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:18:29,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:18:29,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166218 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:18:34,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:18:34,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143939 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:18:39,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:18:39,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00773735 s, 529 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:18:44,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:18:44,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00501833 s, 816 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:18:49,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:18:49,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138538 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:18:54,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:18:54,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766114 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:18:59,837::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:18:59,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0213565 s, 192 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:19:04,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:19:04,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142913 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:19:09,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:19:09,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104246 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:19:14,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:19:14,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851289 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:19:19,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:19:19,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143492 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14520::WARNING::2013-03-17 19:19:23,631::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:19:23,635::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:19:25,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:19:25,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157076 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:19:29,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:19:30,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830799 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:19:33,637::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:19:33,639::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:19:34,047::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:19:34,054::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:19:34,055::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:19:35,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:19:35,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00995158 s, 412 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:19:40,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:19:40,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000669893 s, 6.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:19:45,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:19:45,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130054 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:19:50,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:19:50,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746968 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:19:55,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:19:55,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135654 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:20:00,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:20:00,132::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151189 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:20:05,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:20:05,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141453 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:20:10,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:20:10,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835398 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:20:15,467::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 19:20:15,469::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 19:20:15,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:20:15,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000760801 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:20:15,498::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000760801 s, 2.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 19:20:15,499::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 19:20:15,499::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 19:20:15,501::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 19:20:15,502::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:20:15,502::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:20:15,503::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:20:15,503::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:20:15,504::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:20:15,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:20:15,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:20:15,933::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:20:15,934::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 19:20:15,936::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 19:20:15,938::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 19:20:15,938::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 19:20:15,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:20:16,402::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:20:16,405::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:20:16,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139114 s, 294 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:20:20,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:20:20,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010103 s, 405 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:20:26,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:20:26,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013583 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:20:30,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:20:30,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149758 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14521::WARNING::2013-03-17 19:20:34,081::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:20:34,085::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:20:36,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:20:36,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763747 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:20:40,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:20:40,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824136 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:20:44,086::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 19:20:44,088::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:20:44,089::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:20:44,559::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:20:44,566::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:20:44,567::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:20:46,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:20:46,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194161 s, 211 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:20:50,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:20:50,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902659 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:20:56,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:20:56,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822255 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:21:00,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:21:00,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143844 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:21:06,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:21:06,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802389 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:21:10,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:21:10,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880193 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:21:16,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:21:16,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083597 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:21:20,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:21:20,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159235 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:21:26,959::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:21:26,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815863 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:21:30,465::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:21:30,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147387 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:21:37,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:21:37,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014126 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:21:40,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:21:40,552::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012597 s, 3.3 MB/s\n'; <rc> = 0
>Thread-14522::WARNING::2013-03-17 19:21:44,595::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:21:44,599::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:21:47,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:21:47,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0092379 s, 443 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:21:50,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:21:50,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854522 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:21:54,601::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:21:54,602::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:21:55,064::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:21:55,071::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:21:55,072::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:21:57,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:21:57,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168526 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:22:00,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:22:00,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075296 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:22:07,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:22:07,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861185 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:22:10,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:22:10,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000933607 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:22:17,352::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:22:17,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000978087 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:22:20,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:22:20,712::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139985 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:22:27,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:22:27,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783451 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:22:30,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:22:30,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870181 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:22:37,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:22:37,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741671 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:22:40,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:22:40,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140769 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:22:47,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:22:47,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017542 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:22:50,804::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 19:22:50,806::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 19:22:50,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:22:50,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00133319 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:22:50,847::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00133319 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 19:22:50,848::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 19:22:50,848::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 19:22:50,852::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 19:22:50,853::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:22:50,854::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:22:50,854::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:22:50,854::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:22:50,855::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:22:50,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:22:51,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:22:51,305::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:22:51,307::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 19:22:51,311::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 19:22:51,312::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 19:22:51,312::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 19:22:51,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:22:51,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:22:51,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:22:51,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178461 s, 230 kB/s\n'; <rc> = 0
>Thread-14523::WARNING::2013-03-17 19:22:55,100::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:22:55,104::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:22:57,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:22:57,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0028881 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:23:01,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:23:01,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109755 s, 3.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:23:05,106::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:23:05,108::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:23:05,502::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:23:05,509::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:23:05,511::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:23:07,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:23:07,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016035 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:23:11,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:23:11,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000928267 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:23:17,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:23:17,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843642 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:23:21,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:23:21,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000745719 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:23:27,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:23:27,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013214 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:23:31,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:23:31,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00243727 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:23:37,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:23:37,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793694 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:23:41,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:23:41,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145385 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:23:48,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:23:48,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000947604 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:23:51,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:23:51,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143411 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:23:58,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:23:58,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001292 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:24:02,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:24:02,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076885 s, 5.3 MB/s\n'; <rc> = 0
>Thread-14524::WARNING::2013-03-17 19:24:05,546::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:24:05,549::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:24:08,209::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:24:08,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840426 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:24:12,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:24:12,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144976 s, 2.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:24:15,552::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:24:15,553::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:24:16,024::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:24:16,031::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:24:16,032::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:24:18,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:24:18,331::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00203668 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:24:22,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:24:22,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120955 s, 339 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:24:28,372::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:24:28,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00259849 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:24:32,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:24:32,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789011 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:24:38,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:24:38,484::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00240342 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:24:42,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:24:42,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00989002 s, 414 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:24:48,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:24:48,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760786 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:24:52,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:24:52,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141432 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:24:58,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:24:58,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000732069 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:25:02,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:25:02,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152996 s, 268 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:25:08,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:25:08,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0007932 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:25:12,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:25:12,481::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767295 s, 5.3 MB/s\n'; <rc> = 0
>Thread-14525::WARNING::2013-03-17 19:25:16,060::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:25:16,064::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:25:18,738::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 19:25:18,739::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 19:25:18,740::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:25:18,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00181275 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:25:18,776::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00181275 s, 1.1 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 19:25:18,777::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 19:25:18,777::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 19:25:18,779::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 19:25:18,781::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:25:18,781::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:25:18,782::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:25:18,782::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:25:18,783::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:25:18,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:25:19,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:25:19,197::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:25:19,198::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 19:25:19,203::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 19:25:19,204::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 19:25:19,205::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 19:25:19,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:25:19,569::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:25:19,571::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:25:19,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00682068 s, 601 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:25:22,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:25:22,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00970735 s, 422 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:25:26,072::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:25:26,074::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:25:26,517::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:25:26,524::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:25:26,525::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:25:29,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:25:29,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915455 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:25:32,532::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:25:32,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159562 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:25:39,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:25:39,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760303 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:25:42,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:25:42,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108212 s, 379 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:25:49,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:25:49,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162982 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:25:52,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:25:52,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000903723 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:25:59,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:25:59,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169164 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:26:02,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:26:02,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138575 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:26:09,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:26:09,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136856 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:26:12,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:26:12,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774988 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:26:20,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:26:20,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141592 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:26:22,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:26:22,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138746 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14526::WARNING::2013-03-17 19:26:26,564::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:26:26,568::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:26:30,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:26:30,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824519 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:26:32,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:26:32,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193064 s, 212 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:26:36,577::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 19:26:36,579::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:26:36,580::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:26:37,016::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:26:37,022::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:26:37,023::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:26:40,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:26:40,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167352 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:26:42,854::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:26:42,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150493 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:26:50,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:26:50,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838176 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:26:52,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:26:52,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794384 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:27:00,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:27:00,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755841 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:27:02,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:27:02,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872687 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:27:10,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:27:10,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156781 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:27:12,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:27:13,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765608 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:27:20,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:27:20,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150998 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:27:23,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:27:23,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130451 s, 314 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:27:30,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:27:30,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151889 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:27:33,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:27:33,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074936 s, 5.5 MB/s\n'; <rc> = 0
>Thread-14527::WARNING::2013-03-17 19:27:37,054::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:27:37,058::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:27:40,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:27:40,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846616 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:27:43,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:27:43,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000921453 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:27:47,060::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:27:47,062::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:27:47,524::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:27:47,531::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:27:47,532::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:27:50,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:27:50,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137014 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:27:53,165::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 19:27:53,166::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 19:27:53,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:27:53,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0148559 s, 138 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:27:53,218::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0148559 s, 138 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 19:27:53,218::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 19:27:53,219::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 19:27:53,220::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 19:27:53,221::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:27:53,222::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:27:53,223::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:27:53,223::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:27:53,223::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:27:53,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:27:53,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:27:53,637::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:27:53,638::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 19:27:53,640::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 19:27:53,641::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 19:27:53,642::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 19:27:53,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:27:54,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:27:54,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:27:54,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167124 s, 245 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:28:00,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:28:00,853::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783952 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:28:04,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:28:04,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101581 s, 403 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:28:10,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:28:10,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832582 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:28:14,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:28:14,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151257 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:28:20,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:28:21,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085285 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:28:24,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:28:24,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107653 s, 380 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:28:31,045::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:28:31,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779858 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:28:34,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:28:34,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152634 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:28:41,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:28:41,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144713 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:28:44,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:28:44,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899918 s, 4.6 MB/s\n'; <rc> = 0
>Thread-14528::WARNING::2013-03-17 19:28:47,559::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:28:47,563::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:28:51,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:28:51,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812162 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:28:54,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:28:54,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747621 s, 5.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:28:57,565::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:28:57,566::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:28:58,011::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:28:58,018::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:28:58,019::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:29:01,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:29:01,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000995794 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:29:04,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:29:04,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107641 s, 381 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:29:11,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:29:11,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174992 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:29:14,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:29:14,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118364 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:29:21,416::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:29:21,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790612 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:29:24,429::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:29:24,491::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113974 s, 359 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:29:31,490::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:29:31,517::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163605 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:29:34,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:29:34,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741365 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:29:41,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:29:41,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761547 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:29:44,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:29:44,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159768 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:29:51,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:29:51,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156166 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:29:54,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:29:54,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153667 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14529::WARNING::2013-03-17 19:29:58,054::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:29:58,057::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:30:01,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:30:01,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00214882 s, 1.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:30:04,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:30:04,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109736 s, 373 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:30:08,060::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:30:08,062::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:30:08,517::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:30:08,524::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:30:08,526::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:30:11,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:30:11,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122088 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:30:14,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:30:14,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814127 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:30:21,856::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 19:30:21,857::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 19:30:21,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:30:21,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0015039 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:30:21,904::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0015039 s, 1.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 19:30:21,905::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 19:30:21,905::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 19:30:21,907::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 19:30:21,907::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:30:21,908::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:30:21,909::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:30:21,909::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:30:21,909::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:30:21,910::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:30:22,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:30:22,335::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:30:22,336::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 19:30:22,339::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 19:30:22,340::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 19:30:22,340::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 19:30:22,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:30:22,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:30:22,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:30:22,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162136 s, 253 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:30:24,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:30:24,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163313 s, 251 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:30:32,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:30:32,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816193 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:30:34,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:30:34,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191466 s, 214 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:30:42,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:30:42,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738655 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:30:44,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:30:44,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149686 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:30:52,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:30:53,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748149 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:30:54,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:30:54,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150638 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:31:03,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:31:03,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166153 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:31:04,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:31:04,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146217 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14530::WARNING::2013-03-17 19:31:08,553::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:31:08,557::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:31:13,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:31:13,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810517 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:31:14,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:31:15,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877088 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:31:18,559::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:31:18,560::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:31:18,998::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:31:19,005::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:31:19,006::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:31:23,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:31:23,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812128 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:31:25,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:31:25,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154911 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:31:33,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:31:33,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143385 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:31:35,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:31:35,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135023 s, 303 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:31:43,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:31:43,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166182 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:31:45,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:31:45,140::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865163 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:31:53,455::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:31:53,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143836 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:31:55,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:31:55,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157073 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:32:03,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:32:03,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823945 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:32:05,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:32:05,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145697 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:32:13,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:32:13,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00288511 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:32:15,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:32:15,287::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825464 s, 5.0 MB/s\n'; <rc> = 0
>Thread-14531::WARNING::2013-03-17 19:32:19,051::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:32:19,055::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:32:23,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:32:23,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786818 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:32:25,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:32:25,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796695 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:32:29,057::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 19:32:29,058::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:32:29,059::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:32:29,491::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:32:29,499::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:32:29,501::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:32:33,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:32:33,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162425 s, 252 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:32:35,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:32:35,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133455 s, 307 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:32:43,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:32:43,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145002 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:32:45,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:32:45,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150994 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:32:53,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:32:53,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877966 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:32:55,425::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 19:32:55,427::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 19:32:55,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:32:55,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0115643 s, 177 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:32:55,468::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0115643 s, 177 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 19:32:55,469::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 19:32:55,469::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 19:32:55,471::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 19:32:55,473::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:32:55,474::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:32:55,474::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:32:55,474::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:32:55,475::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:32:55,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:32:55,899::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:32:55,904::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:32:55,905::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 19:32:55,907::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 19:32:55,909::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 19:32:55,909::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 19:32:55,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:32:56,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:32:56,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:32:56,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00950198 s, 431 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:33:04,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:33:04,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00906569 s, 452 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:33:06,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:33:06,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084511 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:33:14,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:33:14,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145408 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:33:16,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:33:16,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183881 s, 223 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:33:24,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:33:24,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00918285 s, 446 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:33:26,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:33:26,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00390288 s, 1.0 MB/s\n'; <rc> = 0
>Thread-14532::WARNING::2013-03-17 19:33:29,529::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:33:29,534::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:33:34,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:33:34,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843937 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:33:36,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:33:36,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0048669 s, 842 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:33:39,536::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:33:39,537::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:33:39,981::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:33:39,988::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:33:39,989::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:33:44,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:33:44,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110121 s, 372 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:33:46,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:33:46,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00094012 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:33:54,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:33:54,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805549 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:33:56,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:33:56,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137315 s, 298 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:34:04,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:34:04,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088995 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:34:06,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:34:06,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156911 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:34:14,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:34:14,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105498 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:34:16,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:34:16,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120081 s, 341 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:34:24,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:34:24,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923812 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:34:26,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:34:26,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014522 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:34:34,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:34:34,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138762 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:34:36,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:34:36,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00775067 s, 528 kB/s\n'; <rc> = 0
>Thread-14533::WARNING::2013-03-17 19:34:40,016::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:34:40,019::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:34:44,778::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:34:44,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166802 s, 246 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:34:46,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:34:46,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145749 s, 2.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:34:50,021::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:34:50,023::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:34:50,482::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:34:50,488::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:34:50,489::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:34:54,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:34:54,899::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000707782 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:34:56,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:34:56,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789579 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:35:04,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:35:04,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104266 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:35:06,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:35:06,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015411 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:35:15,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:35:15,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814921 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:35:16,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:35:16,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868602 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:35:25,097::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 19:35:25,098::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 19:35:25,099::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:35:25,130::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00134923 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:35:25,131::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00134923 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 19:35:25,132::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 19:35:25,132::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 19:35:25,134::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 19:35:25,135::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:35:25,136::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:35:25,136::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:35:25,136::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:35:25,137::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:35:25,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:35:25,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:35:25,561::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:35:25,562::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 19:35:25,565::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 19:35:25,566::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 19:35:25,567::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 19:35:25,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:35:25,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:35:25,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:35:25,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00712165 s, 575 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:35:26,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:35:27,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181561 s, 226 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:35:36,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:35:36,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156356 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:35:37,048::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:35:37,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148787 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:35:46,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:35:46,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765804 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:35:47,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:35:47,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169245 s, 2.4 MB/s\n'; <rc> = 0
>Thread-14534::WARNING::2013-03-17 19:35:50,518::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:35:50,521::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:35:56,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:35:56,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779664 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:35:57,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:35:57,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000693955 s, 5.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:36:00,523::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:36:00,524::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:36:00,938::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:36:00,944::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:36:00,946::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:36:06,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:36:06,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794761 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:36:07,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:36:07,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0289019 s, 142 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:36:16,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:36:16,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131842 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:36:17,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:36:17,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000928154 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:36:26,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:36:26,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896179 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:36:27,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:36:27,331::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836987 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:36:36,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:36:36,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131554 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:36:37,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:36:37,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013193 s, 310 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:36:46,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:36:46,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145294 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:36:47,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:36:47,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129428 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:36:56,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:36:56,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000969216 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:36:57,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:36:57,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122166 s, 3.4 MB/s\n'; <rc> = 0
>Thread-14535::WARNING::2013-03-17 19:37:00,971::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:37:00,974::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:37:06,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:37:06,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106296 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:37:07,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:37:07,513::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079136 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:37:10,983::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:37:10,985::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:37:11,434::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:37:11,440::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:37:11,442::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:37:16,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:37:16,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00070802 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:37:17,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:37:17,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909055 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:37:26,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:37:26,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801851 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:37:27,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:37:27,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077665 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:37:36,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:37:36,958::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000726651 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:37:37,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:37:37,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146447 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:37:46,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:37:47,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801036 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:37:47,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:37:47,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108959 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:37:57,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:37:57,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144959 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:37:57,673::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 19:37:57,675::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 19:37:57,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:37:57,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00202719 s, 1.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:37:57,708::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00202719 s, 1.0 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 19:37:57,708::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 19:37:57,708::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 19:37:57,710::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 19:37:57,711::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:37:57,712::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:37:57,712::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:37:57,713::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:37:57,713::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:37:57,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:37:58,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:37:58,146::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:37:58,147::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 19:37:58,149::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 19:37:58,150::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 19:37:58,151::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 19:37:58,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:37:58,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:37:58,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:37:58,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0026869 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:38:07,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:38:07,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793406 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:38:08,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:38:08,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000993259 s, 4.1 MB/s\n'; <rc> = 0
>Thread-14536::WARNING::2013-03-17 19:38:11,471::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:38:11,477::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:38:17,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:38:17,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076165 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:38:18,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:38:18,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000918082 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:38:21,479::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 19:38:21,480::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:38:21,481::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:38:21,892::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:38:21,900::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:38:21,901::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:38:27,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:38:27,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850324 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:38:28,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:38:28,744::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130307 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:38:37,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:38:37,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846523 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:38:38,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:38:38,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00522325 s, 784 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:38:47,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:38:47,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00210375 s, 1.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:38:48,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:38:48,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835072 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:38:57,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:38:57,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847692 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:38:58,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:38:58,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829105 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:39:07,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:39:07,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774281 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:39:08,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:39:08,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0019493 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:39:17,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:39:17,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016762 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:39:18,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:39:18,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132536 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14537::WARNING::2013-03-17 19:39:21,929::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:39:21,934::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:39:27,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:39:27,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760171 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:39:28,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:39:28,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125558 s, 3.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:39:31,936::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:39:31,937::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:39:32,370::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:39:32,377::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:39:32,379::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:39:37,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:39:37,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087614 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:39:38,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:39:39,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091741 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:39:47,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:39:47,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848496 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:39:49,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:39:49,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079777 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:39:57,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:39:58,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169257 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:39:59,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:39:59,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819435 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:40:08,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:40:08,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187907 s, 218 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:40:09,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:40:09,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740968 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:40:18,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:40:18,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851665 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:40:19,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:40:19,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148774 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:40:28,245::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 19:40:28,248::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 19:40:28,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:40:28,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00136407 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:40:28,282::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00136407 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 19:40:28,283::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 19:40:28,283::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 19:40:28,285::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 19:40:28,286::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:40:28,287::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:40:28,288::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:40:28,288::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:40:28,289::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:40:28,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:40:28,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:40:28,726::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:40:28,727::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 19:40:28,730::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 19:40:28,732::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 19:40:28,733::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 19:40:28,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:40:29,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:40:29,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:40:29,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108597 s, 377 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:40:29,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:40:29,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00654493 s, 626 kB/s\n'; <rc> = 0
>Thread-14538::WARNING::2013-03-17 19:40:32,412::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:40:32,416::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:40:39,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:40:39,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000954554 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:40:39,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:40:39,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174492 s, 2.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:40:42,418::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:40:42,419::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:40:42,862::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:40:42,869::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:40:42,871::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:40:49,305::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:40:49,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:40:49,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00771375 s, 531 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:40:49,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000688811 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:40:59,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:40:59,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:40:59,562::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128871 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:40:59,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155439 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:41:09,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:41:09,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:41:09,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779802 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:41:09,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168584 s, 243 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:41:19,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:41:19,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:41:19,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105648 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:41:19,837::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00348847 s, 1.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:41:29,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:41:29,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:41:29,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00231448 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:41:29,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113108 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:41:39,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:41:40,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:41:40,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00284558 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:41:40,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123712 s, 331 kB/s\n'; <rc> = 0
>Thread-14539::WARNING::2013-03-17 19:41:42,897::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:41:42,901::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 19:41:50,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:41:50,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:41:50,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113485 s, 361 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:41:50,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00278527 s, 1.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:41:52,911::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:41:52,912::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:41:53,408::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:41:53,415::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:41:53,416::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:42:00,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:42:00,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:42:00,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00180726 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:42:00,309::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117962 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:42:10,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:42:10,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:42:10,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113575 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:42:10,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000914387 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:42:20,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:42:20,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:42:20,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131228 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:42:20,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106267 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:42:30,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:42:30,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:42:30,671::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862557 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:42:30,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101038 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:42:40,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:42:40,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:42:40,837::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141145 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:42:40,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00354547 s, 1.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:42:50,861::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:42:50,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:42:50,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149886 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:42:50,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106826 s, 3.8 MB/s\n'; <rc> = 0
>Thread-14540::WARNING::2013-03-17 19:42:53,442::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:42:53,445::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 19:43:00,984::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 19:43:00,985::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 19:43:00,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:43:01,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000805825 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:43:01,017::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000805825 s, 2.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 19:43:01,017::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 19:43:01,018::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 19:43:01,019::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 19:43:01,020::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:43:01,021::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:43:01,021::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:43:01,022::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:43:01,022::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:43:01,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:43:01,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:43:01,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125305 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:43:01,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:43:01,480::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:43:01,481::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 19:43:01,483::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 19:43:01,484::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 19:43:01,485::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 19:43:01,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:43:01,843::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:43:01,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:43:01,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0409648 s, 100 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:43:03,448::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:43:03,450::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:43:03,858::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:43:03,865::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:43:03,867::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:43:11,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:43:11,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822044 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:43:11,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:43:11,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150243 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:43:21,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:43:21,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100144 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:43:21,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:43:21,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800471 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:43:31,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:43:31,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773171 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:43:32,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:43:32,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153021 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:43:41,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:43:41,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756811 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:43:42,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:43:42,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001274 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:43:51,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:43:51,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148359 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:43:52,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:43:52,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838885 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:44:01,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:44:01,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109492 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:44:02,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:44:02,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00928907 s, 441 kB/s\n'; <rc> = 0
>Thread-14541::WARNING::2013-03-17 19:44:03,893::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:44:03,897::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:44:11,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:44:11,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784692 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:44:12,312::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:44:12,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881375 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:44:13,899::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 19:44:13,900::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:44:13,902::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:44:14,367::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:44:14,374::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:44:14,375::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:44:21,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:44:21,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008095 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:44:22,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:44:22,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899966 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:44:31,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:44:31,765::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823074 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:44:32,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:44:32,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138389 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:44:41,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:44:41,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133595 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:44:42,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:44:42,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135512 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:44:51,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:44:51,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00204921 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:44:52,481::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:44:52,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00314492 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:45:01,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:45:01,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814631 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:45:02,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:45:02,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171442 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:45:12,033::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:45:12,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105499 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:45:12,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:45:12,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138097 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14542::WARNING::2013-03-17 19:45:14,413::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:45:14,417::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:45:22,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:45:22,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163865 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:45:22,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:45:22,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855196 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:45:24,420::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:45:24,421::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:45:24,827::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:45:24,834::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:45:24,835::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:45:32,210::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 19:45:32,213::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 19:45:32,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:45:32,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000855159 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:45:32,244::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000855159 s, 2.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 19:45:32,245::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 19:45:32,245::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 19:45:32,247::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 19:45:32,248::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:45:32,249::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:45:32,250::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:45:32,250::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:45:32,251::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:45:32,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:45:32,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:45:32,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:45:32,701::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:45:32,703::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 19:45:32,717::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 19:45:32,721::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 19:45:32,723::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 19:45:32,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104847 s, 391 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:45:32,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:45:33,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:45:33,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:45:33,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182663 s, 224 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:45:42,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:45:42,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015566 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:45:43,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:45:43,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112249 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:45:52,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:45:52,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00249563 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:45:53,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:45:53,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936621 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:46:02,854::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:46:02,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759732 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:46:03,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:46:03,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016526 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:46:12,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:46:12,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136796 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:46:13,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:46:13,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001479 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:46:22,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:46:22,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081946 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:46:23,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:46:23,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119956 s, 341 kB/s\n'; <rc> = 0
>Thread-14543::WARNING::2013-03-17 19:46:24,874::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:46:24,877::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 19:46:32,993::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:46:33,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850188 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:46:34,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:46:34,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900398 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:46:34,884::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:46:34,886::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:46:35,331::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:46:35,338::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:46:35,339::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:46:43,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:46:43,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187258 s, 219 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:46:44,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:46:44,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00271399 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:46:53,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:46:53,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791309 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:46:54,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:46:54,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843006 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:47:03,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:47:03,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806388 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:47:04,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:47:04,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141764 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:47:13,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:47:13,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140185 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:47:14,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:47:14,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000905486 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:47:23,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:47:23,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157357 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:47:24,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:47:24,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881079 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:47:33,260::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:47:33,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808172 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:47:34,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:47:34,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0369561 s, 111 kB/s\n'; <rc> = 0
>Thread-14544::WARNING::2013-03-17 19:47:35,371::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:47:35,377::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 19:47:43,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:47:43,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147567 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:47:45,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:47:45,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00565226 s, 725 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:47:45,380::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:47:45,382::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:47:45,796::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:47:45,803::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:47:45,804::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:47:53,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:47:53,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085976 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:47:55,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:47:55,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000905261 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:48:03,397::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 19:48:03,398::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 19:48:03,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:48:03,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000686838 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:48:03,456::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000686838 s, 3.0 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 19:48:03,457::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 19:48:03,457::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 19:48:03,459::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 19:48:03,460::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:48:03,460::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:48:03,461::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:48:03,461::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:48:03,462::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:48:03,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:48:03,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:48:03,876::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:48:03,877::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 19:48:03,880::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 19:48:03,881::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 19:48:03,881::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 19:48:03,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:48:04,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:48:04,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:48:04,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179009 s, 229 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:48:05,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:48:05,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105131 s, 390 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:48:14,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:48:14,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156639 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:48:15,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:48:15,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074363 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:48:24,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:48:24,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106883 s, 383 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:48:25,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:48:25,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149802 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:48:34,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:48:34,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146744 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:48:35,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:48:35,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167825 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:48:44,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:48:44,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865648 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:48:45,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:48:45,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746509 s, 5.5 MB/s\n'; <rc> = 0
>Thread-14545::WARNING::2013-03-17 19:48:45,846::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:48:45,856::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 19:48:54,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:48:54,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015846 s, 2.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:48:55,860::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:48:55,862::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:48:55,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:48:55,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141165 s, 290 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:48:56,324::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:48:56,330::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:48:56,331::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:49:04,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:49:04,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149216 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:49:05,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:49:06,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821004 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:49:14,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:49:14,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882357 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:49:16,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:49:16,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159286 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:49:24,899::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:49:24,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925627 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:49:26,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:49:26,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877519 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:49:34,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:49:34,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766301 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:49:36,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:49:36,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135967 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:49:44,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:49:45,010::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149942 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:49:46,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:49:46,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160019 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:49:55,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:49:55,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145857 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14546::WARNING::2013-03-17 19:49:56,361::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:49:56,372::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:49:56,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:49:56,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084331 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:50:05,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:50:05,130::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900297 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:50:06,374::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 19:50:06,375::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:50:06,376::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:50:06,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:50:06,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135205 s, 303 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:50:06,825::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:50:06,830::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:50:06,832::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:50:15,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:50:15,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000947284 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:50:16,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:50:16,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813449 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:50:25,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:50:25,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145459 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:50:26,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:50:26,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00274041 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:50:35,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:50:35,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00295284 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:50:36,711::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 19:50:36,712::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 19:50:36,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:50:36,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00161669 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:50:36,760::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00161669 s, 1.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 19:50:36,761::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 19:50:36,761::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 19:50:36,763::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 19:50:36,764::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:50:36,765::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:50:36,765::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:50:36,766::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:50:36,767::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:50:36,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:50:37,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:50:37,219::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:50:37,220::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 19:50:37,223::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 19:50:37,224::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 19:50:37,224::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 19:50:37,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:50:37,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:50:37,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:50:37,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00667883 s, 613 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:50:45,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:50:45,297::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138961 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:50:47,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:50:47,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829619 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:50:55,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:50:55,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790574 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:50:57,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:50:57,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837093 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:51:05,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:51:05,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107019 s, 3.8 MB/s\n'; <rc> = 0
>Thread-14547::WARNING::2013-03-17 19:51:06,856::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:51:06,860::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:51:07,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:51:07,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000935548 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:51:15,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:51:15,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774352 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:51:16,862::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:51:16,864::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:51:17,282::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:51:17,289::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:51:17,290::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:51:17,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:51:17,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182499 s, 224 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:51:25,482::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:51:25,512::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144168 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:51:28,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:51:28,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167119 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:51:35,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:51:35,548::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000963447 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:51:38,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:51:38,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000727574 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:51:45,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:51:45,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126149 s, 325 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:51:48,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:51:48,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156721 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:51:55,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:51:55,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795716 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:51:58,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:51:58,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132745 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:52:05,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:52:05,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152768 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:52:08,350::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:52:08,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166369 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:52:15,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:52:15,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164834 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14548::WARNING::2013-03-17 19:52:17,318::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:52:17,324::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:52:18,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:52:18,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872231 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:52:25,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:52:25,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086271 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:52:27,328::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:52:27,329::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:52:27,745::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:52:27,752::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:52:27,753::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:52:28,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:52:28,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138388 s, 296 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:52:35,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:52:35,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000945696 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:52:38,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:52:38,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089768 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:52:45,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:52:45,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159311 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:52:48,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:52:48,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100735 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:52:55,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:52:55,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000942989 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:52:58,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:52:58,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842526 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:53:05,877::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 19:53:05,878::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 19:53:05,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:53:05,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00138699 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:53:05,924::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00138699 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 19:53:05,925::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 19:53:05,926::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 19:53:05,927::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 19:53:05,928::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:53:05,929::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:53:05,929::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:53:05,930::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:53:05,930::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:53:05,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:53:06,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:53:06,372::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:53:06,373::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 19:53:06,375::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 19:53:06,376::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 19:53:06,377::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 19:53:06,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:53:06,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:53:06,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:53:06,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0201359 s, 203 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:53:08,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:53:08,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187102 s, 219 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:53:16,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:53:16,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151338 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:53:18,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:53:18,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167565 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:53:26,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:53:27,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160021 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14549::WARNING::2013-03-17 19:53:27,781::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:53:27,785::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:53:29,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:53:29,040::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158655 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:53:37,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:53:37,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827374 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:53:37,788::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:53:37,790::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:53:38,201::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:53:38,208::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:53:38,209::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:53:39,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:53:39,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145777 s, 281 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:53:47,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:53:47,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081259 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:53:49,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:53:49,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802057 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:53:57,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:53:57,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166027 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:53:59,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:53:59,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783162 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:54:07,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:54:07,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879542 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:54:09,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:54:09,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104868 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:54:17,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:54:17,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:54:19,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:54:19,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075331 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:54:27,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:54:27,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000657133 s, 6.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:54:29,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:54:29,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000974638 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:54:37,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:54:37,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185849 s, 2.2 MB/s\n'; <rc> = 0
>Thread-14550::WARNING::2013-03-17 19:54:38,236::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:54:38,240::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:54:39,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:54:39,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00180557 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:54:47,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:54:47,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146553 s, 2.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:54:48,249::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:54:48,250::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:54:48,673::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:54:48,680::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:54:48,681::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:54:49,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:54:49,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135953 s, 301 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:54:57,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:54:57,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010042 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:54:59,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:54:59,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103043 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:55:07,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:55:07,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130595 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:55:09,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:55:09,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155934 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:55:17,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:55:17,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898706 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:55:19,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:55:19,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00192732 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:55:27,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:55:27,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769129 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:55:29,993::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:55:30,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146232 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:55:37,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:55:37,733::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752804 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:55:40,076::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 19:55:40,081::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 19:55:40,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:55:40,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00106768 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:55:40,115::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00106768 s, 1.9 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 19:55:40,115::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 19:55:40,116::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 19:55:40,117::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 19:55:40,118::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:55:40,119::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:55:40,119::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:55:40,120::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:55:40,120::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 19:55:40,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:55:40,552::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:55:40,556::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:55:40,557::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 19:55:40,562::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 19:55:40,563::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 19:55:40,564::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 19:55:40,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:55:40,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:55:40,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:55:41,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0083948 s, 488 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:55:47,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:55:47,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0345859 s, 118 kB/s\n'; <rc> = 0
>Thread-14551::WARNING::2013-03-17 19:55:48,708::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:55:48,711::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:55:51,045::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:55:51,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744089 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:55:58,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:55:58,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858443 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:55:58,741::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 19:55:58,743::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:55:58,746::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:55:59,180::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:55:59,187::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:55:59,188::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:56:01,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:56:01,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137252 s, 298 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:56:08,079::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:56:08,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000969492 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:56:11,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:56:11,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839709 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:56:18,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:56:18,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915625 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:56:21,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:56:21,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869469 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:56:28,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:56:28,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151637 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:56:31,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:56:31,401::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148055 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:56:38,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:56:38,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817607 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:56:41,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:56:41,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780761 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:56:48,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:56:48,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148129 s, 277 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:56:51,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:56:51,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148875 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:56:58,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:56:58,483::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0217665 s, 188 kB/s\n'; <rc> = 0
>Thread-14552::WARNING::2013-03-17 19:56:59,215::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:56:59,219::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 19:57:01,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:57:01,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00959646 s, 427 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:57:08,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:57:08,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896492 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:57:09,221::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:57:09,222::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:57:09,627::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:57:09,634::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:57:09,636::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:57:11,685::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:57:11,742::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146003 s, 281 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:57:18,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:57:18,591::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816828 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:57:21,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:57:21,809::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000959572 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:57:28,596::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:57:28,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000973217 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:57:31,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:57:31,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015209 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:57:38,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:57:38,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000750201 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:57:41,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:57:41,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115653 s, 354 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:57:48,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:57:48,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128644 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:57:52,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:57:52,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149274 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:57:58,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:57:58,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120842 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:58:02,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:58:02,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168009 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:58:08,747::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 19:58:08,748::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 19:58:08,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:58:08,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0231339 s, 88.5 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:58:08,988::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0231339 s, 88.5 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 19:58:08,989::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 19:58:08,989::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 19:58:08,991::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 19:58:08,992::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:58:09,143::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:58:09,148::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:58:09,149::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 19:58:09,151::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 19:58:09,156::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:58:09,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:58:09,672::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-14553::WARNING::2013-03-17 19:58:09,671::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-22::DEBUG::2013-03-17 19:58:09,681::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-21::ERROR::2013-03-17 19:58:09,688::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::WARNING::2013-03-17 19:58:09,691::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 19:58:09,692::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 19:58:09,692::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 19:58:09,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:58:10,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:58:10,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:58:10,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868591 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:58:12,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:58:12,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148052 s, 277 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:58:19,690::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:58:19,691::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:58:20,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-21::DEBUG::2013-03-17 19:58:20,161::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-22::DEBUG::2013-03-17 19:58:20,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019039 s, 215 kB/s\n'; <rc> = 0
>Thread-21::WARNING::2013-03-17 19:58:20,196::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:58:20,200::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:58:22,271::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:58:22,347::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0317796 s, 129 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:58:30,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:58:30,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000889653 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:58:32,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:58:32,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916946 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:58:40,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:58:40,260::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919435 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:58:42,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:58:42,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000993888 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:58:50,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:58:50,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898295 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:58:52,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:58:52,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851702 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:59:00,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:59:00,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155057 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:59:02,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:59:02,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018856 s, 217 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:59:10,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:59:10,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764232 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:59:12,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:59:12,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147009 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14554::WARNING::2013-03-17 19:59:20,224::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 19:59:20,228::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 19:59:20,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:59:20,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860704 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:59:22,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:59:22,819::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164965 s, 2.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:59:30,239::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 19:59:30,240::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:59:30,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:59:30,490::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011668 s, 351 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 19:59:30,682::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 19:59:30,688::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 19:59:30,689::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 19:59:32,861::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:59:32,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122864 s, 333 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:59:40,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:59:40,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000927874 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:59:42,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:59:42,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827511 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 19:59:50,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 19:59:50,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137273 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 19:59:53,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 19:59:53,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842296 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:00:00,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:00:00,617::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751921 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:00:03,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:00:03,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000977127 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:00:10,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:00:10,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847843 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:00:13,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:00:13,191::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159426 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:00:20,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:00:20,698::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128872 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:00:23,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:00:23,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155358 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:00:30,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-14555::WARNING::2013-03-17 20:00:30,732::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:00:30,738::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:00:30,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00222012 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:00:33,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:00:33,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831041 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:00:40,741::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:00:40,743::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:00:40,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:00:40,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130598 s, 314 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:00:41,217::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:00:41,223::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:00:41,224::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:00:43,386::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 20:00:43,389::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 20:00:43,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:00:43,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0132718 s, 154 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:00:43,438::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0132718 s, 154 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 20:00:43,438::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 20:00:43,439::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 20:00:43,441::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 20:00:43,442::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:00:43,442::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:00:43,443::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:00:43,443::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:00:43,443::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:00:43,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:00:43,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:00:43,849::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:00:43,850::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 20:00:43,853::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 20:00:43,854::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 20:00:43,854::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 20:00:43,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:00:44,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:00:44,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:00:44,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00800882 s, 511 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:00:50,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:00:50,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112435 s, 364 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:00:54,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:00:54,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925874 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:01:00,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:01:00,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874501 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:01:04,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:01:04,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146352 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:01:10,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:01:10,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813329 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:01:14,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:01:14,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785237 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:01:20,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:01:20,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796758 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:01:24,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:01:24,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170233 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:01:30,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:01:31,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147949 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:01:34,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:01:34,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802447 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:01:41,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:01:41,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812741 s, 5.0 MB/s\n'; <rc> = 0
>Thread-14556::WARNING::2013-03-17 20:01:41,258::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:01:41,264::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 20:01:44,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:01:44,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105927 s, 387 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:01:51,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:01:51,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152579 s, 268 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:01:51,279::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 20:01:51,280::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:01:51,282::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:01:51,674::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:01:51,680::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:01:51,682::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:01:54,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:01:54,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0466576 s, 87.8 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:02:01,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:02:01,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741749 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:02:04,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:02:04,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124814 s, 328 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:02:11,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:02:11,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148088 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:02:14,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:02:14,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786668 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:02:21,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:02:21,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015685 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:02:25,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:02:25,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151829 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:02:31,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:02:31,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766314 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:02:35,118::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:02:35,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823138 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:02:41,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:02:41,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0020764 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:02:45,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:02:45,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172663 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:02:51,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:02:51,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812567 s, 5.0 MB/s\n'; <rc> = 0
>Thread-14557::WARNING::2013-03-17 20:02:51,712::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:02:51,719::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 20:02:55,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:02:55,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153547 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:03:01,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:03:01,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194555 s, 211 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:03:01,722::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:03:01,724::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:03:02,204::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:03:02,225::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:03:02,226::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:03:05,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:03:05,395::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177906 s, 230 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:03:11,593::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 20:03:11,595::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 20:03:11,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:03:11,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000836776 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:03:11,634::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000836776 s, 2.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 20:03:11,635::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 20:03:11,635::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 20:03:11,637::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 20:03:11,638::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:03:11,639::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:03:11,640::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:03:11,640::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:03:11,641::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:03:11,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:03:12,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:03:12,057::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:03:12,058::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 20:03:12,060::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 20:03:12,061::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 20:03:12,062::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 20:03:12,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:03:12,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:03:12,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:03:12,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010277 s, 399 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:03:15,439::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:03:15,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0210915 s, 194 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:03:22,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:03:22,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154083 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:03:25,528::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:03:25,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830875 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:03:32,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:03:32,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148342 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:03:35,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:03:35,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165662 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:03:42,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:03:42,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126462 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:03:45,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:03:45,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800518 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:03:52,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:03:52,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138447 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:03:55,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:03:55,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091807 s, 4.5 MB/s\n'; <rc> = 0
>Thread-14558::WARNING::2013-03-17 20:04:02,256::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:04:02,260::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:04:02,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:04:02,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781317 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:04:05,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:04:05,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869052 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:04:12,262::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:04:12,264::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:04:12,691::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-22::DEBUG::2013-03-17 20:04:12,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-21::WARNING::2013-03-17 20:04:12,745::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:04:12,761::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:04:12,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180312 s, 227 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:04:15,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:04:15,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162515 s, 252 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:04:22,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:04:22,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122093 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:04:25,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:04:26,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162123 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:04:32,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:04:32,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074483 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:04:36,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:04:36,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000960834 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:04:42,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:04:42,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170245 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:04:46,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:04:46,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146759 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:04:52,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:04:53,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155508 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:04:56,221::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:04:56,260::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150639 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:05:03,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:05:03,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139373 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:05:06,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:05:06,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152836 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14559::WARNING::2013-03-17 20:05:12,819::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:05:12,823::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:05:13,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:05:13,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776159 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:05:16,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:05:16,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000993727 s, 4.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:05:22,825::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:05:22,826::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:05:23,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:05:23,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178782 s, 229 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:05:23,321::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:05:23,327::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:05:23,328::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:05:26,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:05:26,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172744 s, 237 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:05:33,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:05:33,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860653 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:05:36,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:05:36,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871873 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:05:43,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:05:43,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901863 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:05:46,617::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 20:05:46,618::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 20:05:46,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:05:46,658::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00163589 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:05:46,659::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00163589 s, 1.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 20:05:46,660::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 20:05:46,660::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 20:05:46,661::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 20:05:46,662::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:05:46,663::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:05:46,664::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:05:46,664::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:05:46,664::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:05:46,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:05:47,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:05:47,110::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:05:47,111::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 20:05:47,114::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 20:05:47,115::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 20:05:47,115::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 20:05:47,117::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:05:47,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:05:47,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:05:47,526::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0034381 s, 1.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:05:53,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:05:53,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181355 s, 226 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:05:57,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:05:57,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806639 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:06:03,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:06:03,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193661 s, 212 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:06:07,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:06:07,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00671928 s, 610 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:06:13,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:06:13,392::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145197 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:06:17,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:06:17,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013812 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14560::WARNING::2013-03-17 20:06:23,358::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:06:23,360::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:06:23,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:06:23,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00286778 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:06:27,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:06:27,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754834 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:06:33,366::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:06:33,368::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:06:33,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:06:33,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014935 s, 274 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:06:33,813::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:06:33,819::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:06:33,820::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:06:37,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:06:37,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113381 s, 361 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:06:43,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:06:43,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817507 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:06:47,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:06:47,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000704152 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:06:53,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:06:53,593::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170072 s, 241 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:06:58,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:06:58,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017582 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:07:03,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:07:03,627::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000725942 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:07:08,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:07:08,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107791 s, 380 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:07:13,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:07:13,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139892 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:07:18,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:07:18,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00195254 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:07:23,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:07:23,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015295 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:07:28,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:07:28,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157706 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:07:33,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:07:33,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759391 s, 5.4 MB/s\n'; <rc> = 0
>Thread-14561::WARNING::2013-03-17 20:07:33,846::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:07:33,850::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 20:07:38,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:07:38,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826681 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:07:43,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:07:43,852::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 20:07:43,855::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:07:43,856::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:07:43,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169664 s, 241 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:07:44,333::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:07:44,340::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:07:44,341::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:07:48,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:07:48,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108197 s, 379 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:07:53,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:07:53,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082621 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:07:58,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:07:58,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772421 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:08:03,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:08:03,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080342 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:08:08,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:08:08,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762923 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:08:13,959::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 20:08:13,961::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 20:08:13,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:08:14,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00131896 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:08:14,005::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00131896 s, 1.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 20:08:14,005::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 20:08:14,005::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 20:08:14,007::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 20:08:14,008::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:08:14,009::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:08:14,009::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:08:14,010::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:08:14,010::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:08:14,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:08:14,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:08:14,450::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:08:14,451::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 20:08:14,454::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 20:08:14,455::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 20:08:14,455::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 20:08:14,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:08:14,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:08:14,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:08:14,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135831 s, 302 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:08:18,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:08:18,700::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00513997 s, 797 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:08:24,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:08:24,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145517 s, 281 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:08:28,742::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:08:28,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000686431 s, 6.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:08:34,950::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:08:34,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170961 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:08:38,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:08:38,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141512 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14562::WARNING::2013-03-17 20:08:44,368::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:08:44,372::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:08:45,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:08:45,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00809113 s, 506 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:08:48,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:08:48,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794316 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:08:54,374::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:08:54,376::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:08:54,839::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:08:54,846::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:08:54,847::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:08:55,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:08:55,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148909 s, 275 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:08:58,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:08:59,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00661713 s, 619 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:09:05,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:09:05,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851851 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:09:09,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:09:09,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081916 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:09:15,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:09:15,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143106 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:09:19,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:09:19,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101303 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:09:25,178::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:09:25,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780101 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:09:29,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:09:29,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177479 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:09:35,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:09:35,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843587 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:09:39,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:09:39,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140841 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:09:45,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:09:45,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00811687 s, 505 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:09:49,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:09:49,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014696 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14563::WARNING::2013-03-17 20:09:54,870::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:09:54,874::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:09:55,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:09:55,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0374831 s, 109 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:09:59,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:09:59,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778525 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:10:04,876::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:10:04,878::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:10:05,336::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:10:05,342::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:10:05,344::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:10:05,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:10:05,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019909 s, 206 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:10:09,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:10:09,552::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127691 s, 321 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:10:15,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:10:15,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795819 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:10:19,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:10:19,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000687026 s, 6.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:10:25,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:10:25,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862415 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:10:29,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:10:29,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831983 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:10:35,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:10:35,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128911 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:10:39,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:10:39,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146319 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:10:45,556::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:10:45,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153537 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:10:49,808::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 20:10:49,809::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 20:10:49,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:10:49,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000672139 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:10:49,841::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000672139 s, 3.0 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 20:10:49,841::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 20:10:49,841::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 20:10:49,843::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 20:10:49,844::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:10:49,845::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:10:49,845::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:10:49,846::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:10:49,846::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:10:49,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:10:50,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:10:50,297::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:10:50,298::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 20:10:50,300::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 20:10:50,301::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 20:10:50,302::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 20:10:50,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:10:50,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:10:50,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:10:50,734::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185144 s, 221 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:10:55,606::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:10:55,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173729 s, 236 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:11:00,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:11:00,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00747205 s, 548 kB/s\n'; <rc> = 0
>Thread-14564::WARNING::2013-03-17 20:11:05,406::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:11:05,410::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:11:05,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:11:05,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808095 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:11:10,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:11:10,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149309 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:11:15,412::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:11:15,413::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:11:15,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:11:15,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0350785 s, 117 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:11:15,916::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:11:15,921::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:11:15,925::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:11:20,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:11:21,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0209687 s, 195 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:11:25,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:11:25,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858688 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:11:31,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:11:31,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847048 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:11:35,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:11:35,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150059 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:11:41,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:11:41,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757697 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:11:45,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:11:45,881::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105472 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:11:51,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:11:51,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000680306 s, 6.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:11:55,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:11:55,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0205296 s, 200 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:12:01,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:12:01,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169197 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:12:06,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:12:06,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00556778 s, 736 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:12:11,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:12:11,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172833 s, 237 kB/s\n'; <rc> = 0
>Thread-14565::WARNING::2013-03-17 20:12:15,950::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:12:15,954::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:12:16,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:12:16,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013437 s, 305 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:12:21,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:12:21,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000725042 s, 5.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:12:25,956::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:12:25,957::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:12:26,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:12:26,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011741 s, 349 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:12:26,391::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:12:26,397::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:12:26,398::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:12:31,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:12:31,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151391 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:12:36,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:12:36,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794418 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:12:41,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:12:41,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865152 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:12:46,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:12:46,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00586671 s, 698 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:12:51,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:12:51,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881776 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:12:56,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:12:56,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912563 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:13:01,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:13:01,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802857 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:13:06,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:13:06,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000988017 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:13:11,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:13:11,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179003 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:13:16,499::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 20:13:16,501::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 20:13:16,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:13:16,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00097222 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:13:16,544::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00097222 s, 2.1 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 20:13:16,545::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 20:13:16,545::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 20:13:16,547::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 20:13:16,548::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:13:16,549::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:13:16,549::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:13:16,549::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:13:16,550::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:13:16,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:13:16,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:13:16,988::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:13:16,990::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 20:13:16,993::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 20:13:16,994::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 20:13:16,994::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 20:13:16,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:13:17,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:13:17,429::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:13:17,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126059 s, 325 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:13:21,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:13:21,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0198509 s, 206 kB/s\n'; <rc> = 0
>Thread-14566::WARNING::2013-03-17 20:13:26,423::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:13:26,427::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:13:27,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:13:27,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000723737 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:13:31,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:13:32,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887666 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:13:36,428::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 20:13:36,430::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:13:36,431::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:13:36,843::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:13:36,849::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:13:36,851::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:13:37,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:13:37,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135079 s, 303 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:13:42,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:13:42,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000978456 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:13:47,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:13:47,607::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138933 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:13:52,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:13:52,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00511756 s, 800 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:13:57,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:13:57,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089961 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:14:02,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:14:02,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815964 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:14:07,647::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:14:07,681::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126918 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:14:12,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:14:12,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144894 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:14:17,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:14:17,738::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118068 s, 347 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:14:22,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:14:22,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139342 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:14:27,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:14:27,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186085 s, 220 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:14:32,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:14:32,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783695 s, 5.2 MB/s\n'; <rc> = 0
>Thread-14567::WARNING::2013-03-17 20:14:36,878::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:14:36,883::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:14:37,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:14:37,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00401906 s, 1.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:14:42,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:14:42,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884258 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:14:46,886::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:14:46,887::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:14:47,294::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:14:47,301::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:14:47,302::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:14:47,854::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:14:47,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131992 s, 310 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:14:52,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:14:52,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901162 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:14:57,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:14:57,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125268 s, 327 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:15:02,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:15:02,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848698 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:15:07,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:15:08,040::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000669498 s, 6.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:15:12,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:15:12,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144825 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:15:18,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:15:18,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181036 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:15:22,781::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:15:22,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000725342 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:15:28,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:15:28,163::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783739 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:15:32,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:15:32,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143263 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:15:38,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:15:38,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116891 s, 350 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:15:42,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:15:42,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0019425 s, 2.1 MB/s\n'; <rc> = 0
>Thread-14568::WARNING::2013-03-17 20:15:47,329::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:15:47,333::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:15:48,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:15:48,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788556 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:15:53,033::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 20:15:53,034::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 20:15:53,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:15:53,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00135441 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:15:53,082::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00135441 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 20:15:53,082::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 20:15:53,082::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 20:15:53,084::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 20:15:53,085::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:15:53,086::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:15:53,086::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:15:53,086::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:15:53,087::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:15:53,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:15:53,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:15:53,527::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:15:53,528::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 20:15:53,531::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 20:15:53,532::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 20:15:53,532::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 20:15:53,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:15:53,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:15:53,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:15:53,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00535789 s, 764 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:15:57,336::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:15:57,337::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:15:57,765::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:15:57,772::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:15:57,773::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:15:58,324::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:15:58,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150406 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:16:03,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:16:04,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751557 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:16:08,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:16:08,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000930753 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:16:14,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:16:14,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0198219 s, 207 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:16:18,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:16:18,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804615 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:16:24,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:16:24,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0224092 s, 183 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:16:28,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:16:28,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774311 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:16:34,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:16:34,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142489 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:16:38,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:16:38,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142502 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:16:44,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:16:44,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149692 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:16:48,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:16:48,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163436 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:16:54,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:16:54,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092087 s, 4.4 MB/s\n'; <rc> = 0
>Thread-14569::WARNING::2013-03-17 20:16:57,801::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:16:57,807::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:16:58,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:16:58,636::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870872 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:17:04,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:17:04,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000717921 s, 5.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:17:07,809::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:17:07,811::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:17:08,237::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:17:08,244::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:17:08,245::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:17:08,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:17:08,819::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111378 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:17:14,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:17:14,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143672 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:17:18,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:17:18,890::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192028 s, 213 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:17:24,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:17:24,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.031793 s, 129 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:17:28,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:17:28,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860165 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:17:34,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:17:34,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818655 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:17:38,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:17:38,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126422 s, 324 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:17:44,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:17:44,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000723193 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:17:48,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:17:49,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143516 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:17:54,861::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:17:54,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0086245 s, 475 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:17:59,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:17:59,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142297 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:18:04,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:18:04,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116941 s, 3.5 MB/s\n'; <rc> = 0
>Thread-14570::WARNING::2013-03-17 20:18:08,269::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:18:08,273::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:18:09,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:18:09,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804055 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:18:15,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:18:15,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103116 s, 4.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:18:18,275::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:18:18,276::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:18:18,688::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:18:18,695::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:18:18,697::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:18:19,112::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 20:18:19,115::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 20:18:19,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:18:19,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0121519 s, 169 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:18:19,156::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0121519 s, 169 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 20:18:19,156::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 20:18:19,157::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 20:18:19,159::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 20:18:19,160::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:18:19,161::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:18:19,162::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:18:19,162::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:18:19,163::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:18:19,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:18:19,526::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:18:19,530::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:18:19,531::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 20:18:19,533::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 20:18:19,535::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 20:18:19,535::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 20:18:19,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:18:19,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:18:19,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:18:19,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0066265 s, 618 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:18:25,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:18:25,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744654 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:18:29,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:18:30,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143011 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:18:35,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:18:35,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0310798 s, 132 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:18:40,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:18:40,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828954 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:18:45,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:18:45,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113319 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:18:50,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:18:50,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0318956 s, 128 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:18:55,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:18:55,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143417 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:19:00,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:19:00,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872458 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:19:05,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:19:05,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125514 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:19:10,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:19:10,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839815 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:19:15,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:19:15,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00224676 s, 1.8 MB/s\n'; <rc> = 0
>Thread-14571::WARNING::2013-03-17 20:19:18,724::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:19:18,727::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:19:20,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:19:20,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108098 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:19:25,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:19:25,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101373 s, 4.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:19:28,729::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 20:19:28,730::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:19:28,732::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:19:29,136::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:19:29,152::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:19:29,153::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:19:30,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:19:30,260::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154815 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:19:35,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:19:35,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016115 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:19:40,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:19:40,297::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840408 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:19:45,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:19:45,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870109 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:19:50,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:19:50,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015312 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:19:55,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:19:55,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869168 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:20:00,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:20:00,372::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00215382 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:20:05,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:20:05,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155383 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:20:10,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:20:10,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150384 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:20:16,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:20:16,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145529 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:20:20,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:20:20,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147289 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:20:26,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:20:26,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163005 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14572::WARNING::2013-03-17 20:20:29,191::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:20:29,198::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:20:30,471::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:20:30,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117026 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:20:36,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:20:36,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788809 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:20:39,200::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:20:39,202::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:20:39,633::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:20:39,639::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:20:39,641::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:20:40,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:20:40,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802098 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:20:46,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:20:46,266::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899768 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:20:50,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:20:50,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140126 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:20:56,307::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 20:20:56,308::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 20:20:56,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:20:56,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000765822 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:20:56,335::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000765822 s, 2.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 20:20:56,336::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 20:20:56,336::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 20:20:56,339::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 20:20:56,340::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:20:56,341::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:20:56,342::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:20:56,342::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:20:56,343::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:20:56,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:20:56,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:20:56,775::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:20:56,776::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 20:20:56,778::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 20:20:56,780::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 20:20:56,780::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 20:20:56,781::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:20:57,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:20:57,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:20:57,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923716 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:21:00,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:21:00,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120123 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:21:07,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:21:07,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847306 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:21:10,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:21:10,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146326 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:21:17,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:21:17,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854604 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:21:20,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:21:20,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148116 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:21:27,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:21:27,382::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179694 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:21:30,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:21:30,756::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130323 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:21:37,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:21:37,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000700923 s, 5.8 MB/s\n'; <rc> = 0
>Thread-14573::WARNING::2013-03-17 20:21:39,666::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:21:39,670::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:21:40,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:21:40,818::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0267724 s, 153 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:21:47,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:21:47,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100292 s, 4.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:21:49,672::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:21:49,673::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:21:50,104::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:21:50,116::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:21:50,118::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:21:50,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:21:50,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771796 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:21:57,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:21:57,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756608 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:22:00,859::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:22:00,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793928 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:22:07,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:22:07,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152076 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:22:10,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:22:10,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000695042 s, 5.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:22:17,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:22:17,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014379 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:22:20,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:22:20,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155832 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:22:27,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:22:27,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100363 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:22:31,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:22:31,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110791 s, 370 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:22:37,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:22:37,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160792 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:22:41,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:22:41,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860914 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:22:47,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:22:48,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763744 s, 5.4 MB/s\n'; <rc> = 0
>Thread-14574::WARNING::2013-03-17 20:22:50,150::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:22:50,153::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:22:51,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:22:51,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00473331 s, 865 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:22:58,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:22:58,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170001 s, 2.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:23:00,156::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:23:00,157::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:23:00,558::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:23:00,565::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:23:00,566::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:23:01,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:23:01,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109205 s, 375 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:23:08,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:23:08,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149421 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:23:11,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:23:11,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120332 s, 340 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:23:18,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:23:18,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000951843 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:23:21,239::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 20:23:21,240::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 20:23:21,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:23:21,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000735511 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:23:21,269::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000735511 s, 2.8 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 20:23:21,270::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 20:23:21,270::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 20:23:21,272::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 20:23:21,273::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:23:21,273::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:23:21,274::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:23:21,274::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:23:21,275::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:23:21,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:23:21,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:23:21,713::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:23:21,714::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 20:23:21,717::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 20:23:21,718::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 20:23:21,718::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 20:23:21,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:23:22,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:23:22,171::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:23:22,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169738 s, 241 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:23:28,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:23:28,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740211 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:23:32,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:23:32,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124048 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:23:38,345::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:23:38,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147592 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:23:42,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:23:42,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768458 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:23:48,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:23:48,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173496 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:23:52,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:23:52,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143059 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:23:58,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:23:58,562::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122115 s, 3.4 MB/s\n'; <rc> = 0
>Thread-14575::WARNING::2013-03-17 20:24:00,597::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:24:00,602::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:24:02,350::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:24:02,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165855 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:24:08,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:24:08,636::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899407 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:24:10,605::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:24:10,606::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:24:11,023::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:24:11,029::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:24:11,031::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:24:12,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:24:12,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795637 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:24:18,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:24:18,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897495 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:24:22,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:24:22,478::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0196181 s, 209 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:24:28,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:24:28,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000952221 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:24:32,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:24:32,513::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843764 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:24:38,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:24:38,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00194338 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:24:42,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:24:42,562::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0019854 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:24:48,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:24:48,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000998412 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:24:52,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:24:52,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911609 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:24:58,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:24:59,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140074 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:25:02,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:25:02,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864328 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:25:09,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:25:09,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110454 s, 3.7 MB/s\n'; <rc> = 0
>Thread-14576::WARNING::2013-03-17 20:25:11,071::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:25:11,075::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:25:12,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:25:12,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152996 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:25:19,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:25:19,175::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800685 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:25:21,077::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 20:25:21,078::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:25:21,080::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:25:21,476::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:25:21,483::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:25:21,484::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:25:22,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:25:22,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101152 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:25:29,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:25:29,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078195 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:25:32,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:25:32,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821575 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:25:39,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:25:39,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143596 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:25:42,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:25:42,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00230144 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:25:49,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:25:49,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148686 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:25:52,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:25:52,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141425 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:25:59,461::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 20:25:59,463::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 20:25:59,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:25:59,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000768004 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:25:59,494::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000768004 s, 2.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 20:25:59,494::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 20:25:59,494::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 20:25:59,496::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 20:25:59,501::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:25:59,502::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:25:59,502::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:25:59,502::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:25:59,503::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:25:59,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:25:59,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:25:59,928::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:25:59,929::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 20:25:59,933::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 20:25:59,934::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 20:25:59,935::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 20:25:59,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:26:00,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:26:00,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:26:00,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121184 s, 338 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:26:02,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:26:02,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00569931 s, 719 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:26:10,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:26:10,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154838 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:26:12,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:26:13,054::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00620273 s, 660 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:26:20,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:26:20,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000721387 s, 5.7 MB/s\n'; <rc> = 0
>Thread-14577::WARNING::2013-03-17 20:26:21,516::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:26:21,521::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:26:23,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:26:23,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100776 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:26:30,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:26:30,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842037 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:26:31,525::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:26:31,527::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:26:31,934::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:26:31,941::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:26:31,942::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:26:33,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:26:33,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170418 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:26:40,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:26:40,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136008 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:26:43,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:26:43,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150639 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:26:50,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:26:50,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116463 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:26:53,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:26:53,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122645 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:27:00,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:27:00,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008262 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:27:03,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:27:03,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140956 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:27:10,993::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:27:11,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916682 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:27:13,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:27:13,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00343566 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:27:21,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:27:21,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077858 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:27:23,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:27:23,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171257 s, 239 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:27:31,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:27:31,238::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083715 s, 4.9 MB/s\n'; <rc> = 0
>Thread-14578::WARNING::2013-03-17 20:27:31,977::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:27:31,981::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:27:33,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:27:33,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118327 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:27:41,490::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:27:41,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836501 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:27:41,983::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:27:41,985::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:27:42,428::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:27:42,434::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:27:42,435::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:27:43,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:27:43,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149059 s, 275 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:27:51,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:27:51,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888403 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:27:53,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:27:53,734::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146578 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:28:01,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:28:01,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127576 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:28:03,740::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:28:03,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158362 s, 259 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:28:11,760::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:28:11,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802994 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:28:13,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:28:13,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00232522 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:28:21,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:28:21,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127883 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:28:23,838::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 20:28:23,839::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 20:28:23,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:28:23,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00128739 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:28:23,879::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00128739 s, 1.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 20:28:23,880::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 20:28:23,880::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 20:28:23,881::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 20:28:23,882::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:28:23,883::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:28:23,883::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:28:23,884::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:28:23,884::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:28:23,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:28:24,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:28:24,337::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:28:24,338::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 20:28:24,340::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 20:28:24,341::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 20:28:24,342::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 20:28:24,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:28:24,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:28:24,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:28:24,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162342 s, 252 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:28:31,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:28:31,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000963553 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:28:34,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:28:34,853::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147267 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:28:41,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:28:42,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157253 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14579::WARNING::2013-03-17 20:28:42,473::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:28:42,483::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:28:44,861::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:28:44,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144993 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:28:52,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:28:52,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127903 s, 320 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:28:52,488::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:28:52,489::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:28:52,949::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:28:52,956::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:28:52,957::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:28:54,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:28:54,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778687 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:29:02,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:29:02,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838339 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:29:04,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:29:04,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000929819 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:29:12,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:29:12,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075884 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:29:14,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:29:15,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798994 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:29:22,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:29:22,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813731 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:29:25,019::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:29:25,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018754 s, 218 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:29:32,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:29:32,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765421 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:29:35,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:29:35,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882536 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:29:42,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:29:42,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141082 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:29:45,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:29:45,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147607 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:29:52,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:29:52,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906153 s, 4.5 MB/s\n'; <rc> = 0
>Thread-14580::WARNING::2013-03-17 20:29:52,987::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:29:52,990::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:29:55,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:29:55,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014929 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:30:02,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:30:02,996::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:30:03,001::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:30:03,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103898 s, 394 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:30:03,491::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:30:03,496::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:30:03,498::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:30:05,209::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:30:05,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00201901 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:30:13,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:30:13,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000935819 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:30:15,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:30:15,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870076 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:30:23,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:30:23,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000718397 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:30:25,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:30:25,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153626 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:30:33,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:30:33,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146169 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:30:35,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:30:35,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00090092 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:30:43,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:30:43,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749751 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:30:45,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:30:45,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156451 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:30:53,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:30:53,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760805 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:30:55,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:30:55,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000962829 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:31:03,458::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 20:31:03,460::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 20:31:03,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:31:03,521::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00068538 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14581::WARNING::2013-03-17 20:31:03,528::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-20::DEBUG::2013-03-17 20:31:03,534::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00068538 s, 3.0 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 20:31:03,536::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 20:31:03,540::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-21::ERROR::2013-03-17 20:31:03,544::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 20:31:03,558::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 20:31:03,559::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:31:03,560::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:31:03,560::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:31:03,561::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:31:03,561::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:31:03,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:31:03,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:31:03,995::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:31:03,996::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 20:31:03,999::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 20:31:04,000::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 20:31:04,001::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 20:31:04,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:31:04,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:31:04,372::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:31:04,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00332144 s, 1.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:31:05,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:31:05,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00098386 s, 4.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:31:13,558::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 20:31:13,559::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:31:13,561::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:31:14,000::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:31:14,009::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:31:14,010::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:31:14,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:31:14,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153082 s, 268 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:31:15,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:31:15,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152308 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:31:24,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:31:24,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819168 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:31:25,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:31:25,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00448367 s, 914 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:31:34,612::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:31:34,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785321 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:31:35,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:31:35,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104833 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:31:44,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:31:44,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000986131 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:31:45,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:31:45,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142162 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:31:54,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:31:54,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012591 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:31:55,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:31:55,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153538 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:32:04,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:32:04,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144552 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:32:05,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:32:05,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135071 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14582::WARNING::2013-03-17 20:32:14,047::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:32:14,050::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 20:32:14,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:32:14,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851038 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:32:15,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:32:15,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00534922 s, 766 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:32:24,052::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:32:24,054::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:32:24,531::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:32:24,537::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:32:24,539::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:32:25,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:32:25,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131076 s, 312 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:32:25,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:32:25,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189905 s, 216 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:32:35,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:32:35,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00096833 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:32:35,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:32:35,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923912 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:32:45,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:32:45,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014977 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:32:45,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:32:45,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00388119 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:32:55,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:32:55,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920652 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:32:55,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:32:55,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749734 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:33:05,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:33:05,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154164 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:33:05,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:33:06,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155521 s, 263 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:33:15,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:33:15,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778443 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:33:16,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:33:16,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086005 s, 4.8 MB/s\n'; <rc> = 0
>Thread-14583::WARNING::2013-03-17 20:33:24,568::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:33:24,572::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 20:33:25,471::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:33:25,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000731866 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:33:26,088::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 20:33:26,090::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 20:33:26,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:33:26,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000685303 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:33:26,117::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000685303 s, 3.0 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 20:33:26,117::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 20:33:26,118::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 20:33:26,119::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 20:33:26,120::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:33:26,121::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:33:26,121::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:33:26,122::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:33:26,122::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:33:26,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:33:26,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:33:26,539::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:33:26,540::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 20:33:26,542::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 20:33:26,543::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 20:33:26,544::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 20:33:26,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:33:26,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:33:26,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:33:27,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169213 s, 242 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:33:34,574::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:33:34,575::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:33:34,999::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:33:35,006::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:33:35,008::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:33:35,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:33:35,591::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134042 s, 306 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:33:37,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:33:37,060::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919645 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:33:45,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:33:45,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092169 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:33:47,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:33:47,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757163 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:33:55,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:33:55,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159608 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:33:57,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:33:57,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001437 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:34:05,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:34:05,805::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756668 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:34:07,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:34:07,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141786 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:34:15,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:34:15,890::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175095 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:34:17,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:34:17,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000687524 s, 6.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:34:25,933::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:34:25,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175587 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:34:27,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:34:27,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151215 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14584::WARNING::2013-03-17 20:34:35,034::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:34:35,038::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 20:34:36,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:34:36,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149642 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:34:37,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:34:37,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088235 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:34:45,041::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:34:45,043::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:34:45,508::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:34:45,515::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:34:45,516::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:34:46,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:34:46,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137147 s, 299 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:34:47,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:34:47,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138899 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:34:56,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:34:56,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773658 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:34:57,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:34:57,392::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00090594 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:35:06,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:35:06,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177383 s, 231 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:35:07,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:35:07,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00209775 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:35:16,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:35:16,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147454 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:35:17,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:35:17,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133673 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:35:26,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:35:26,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139409 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:35:27,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:35:27,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141248 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:35:36,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:35:36,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795513 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:35:37,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:35:37,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132699 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14585::WARNING::2013-03-17 20:35:45,541::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:35:45,544::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 20:35:46,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:35:46,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882311 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:35:47,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:35:47,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841034 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:35:55,547::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:35:55,548::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:35:55,999::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:35:56,006::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:35:56,007::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:35:56,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:35:56,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116361 s, 352 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:35:57,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:35:57,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769981 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:36:06,714::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 20:36:06,716::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 20:36:06,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:36:06,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00569969 s, 359 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:36:06,752::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00569969 s, 359 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 20:36:06,753::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 20:36:06,753::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 20:36:06,755::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 20:36:06,756::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:36:06,757::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:36:06,757::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:36:06,758::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:36:06,758::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:36:06,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:36:07,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:36:07,196::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:36:07,198::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 20:36:07,201::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 20:36:07,202::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 20:36:07,203::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 20:36:07,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:36:07,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:36:07,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:36:07,611::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00533351 s, 768 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:36:07,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:36:07,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128094 s, 320 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:36:17,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:36:17,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:36:17,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118427 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:36:17,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143394 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:36:27,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:36:27,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:36:27,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000983556 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:36:27,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118366 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:36:37,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:36:38,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000638528 s, 6.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:36:38,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:36:38,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000985622 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:36:48,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:36:48,060::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171059 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:36:48,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:36:48,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765874 s, 5.3 MB/s\n'; <rc> = 0
>Thread-14586::WARNING::2013-03-17 20:36:56,036::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:36:56,039::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:36:58,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:36:58,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844402 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:36:58,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:36:58,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871291 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:37:06,041::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 20:37:06,042::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:37:06,043::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:37:06,458::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:37:06,465::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:37:06,466::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:37:08,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:37:08,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844478 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:37:08,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:37:08,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181006 s, 226 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:37:18,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:37:18,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00224938 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:37:18,350::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:37:18,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179675 s, 228 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:37:28,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:37:28,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904692 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:37:28,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:37:28,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790548 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:37:38,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:37:38,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00320262 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:37:38,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:37:38,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808578 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:37:48,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:37:48,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894631 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:37:48,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:37:48,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789409 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:37:58,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:37:58,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873398 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:37:58,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:37:58,692::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00794445 s, 516 kB/s\n'; <rc> = 0
>Thread-14587::WARNING::2013-03-17 20:38:06,500::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:38:06,504::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:38:08,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:38:08,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000685883 s, 6.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:38:08,742::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:38:08,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762402 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:38:16,506::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:38:16,507::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:38:16,915::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:38:16,922::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:38:16,924::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:38:18,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:38:18,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126494 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:38:18,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:38:18,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149928 s, 273 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:38:28,424::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 20:38:28,425::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 20:38:28,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:38:28,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00084704 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:38:28,456::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00084704 s, 2.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 20:38:28,456::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 20:38:28,456::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 20:38:28,458::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 20:38:28,465::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:38:28,466::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:38:28,466::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:38:28,467::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:38:28,467::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:38:28,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:38:28,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:38:28,890::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:38:28,891::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-20::DEBUG::2013-03-17 20:38:28,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::WARNING::2013-03-17 20:38:28,902::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 20:38:28,929::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 20:38:28,943::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 20:38:28,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:38:28,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0215679 s, 190 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:38:29,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:38:29,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:38:29,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0157252 s, 260 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:38:39,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:38:39,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152766 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:38:39,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:38:39,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768443 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:38:49,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:38:49,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846704 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:38:49,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:38:49,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830808 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:38:59,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:38:59,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149224 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:38:59,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:38:59,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787518 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:39:09,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:39:09,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166357 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:39:09,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:39:09,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151693 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14588::WARNING::2013-03-17 20:39:16,950::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:39:16,954::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 20:39:19,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:39:19,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807625 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:39:19,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:39:19,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00488203 s, 839 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:39:26,956::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:39:26,958::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:39:27,428::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:39:27,435::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:39:27,436::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:39:29,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:39:29,494::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145214 s, 282 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:39:29,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:39:29,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119534 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:39:39,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:39:39,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00072943 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:39:39,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:39:39,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101252 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:39:49,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:39:49,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159625 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:39:49,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:39:49,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000745366 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:39:59,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:39:59,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110606 s, 370 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:39:59,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:39:59,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100932 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:40:09,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:40:09,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080826 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:40:09,922::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:40:09,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159877 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:40:19,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:40:19,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122877 s, 333 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:40:19,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:40:20,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855002 s, 4.8 MB/s\n'; <rc> = 0
>Thread-14589::WARNING::2013-03-17 20:40:27,461::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:40:27,465::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 20:40:29,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:40:29,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936922 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:40:30,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:40:30,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016719 s, 2.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:40:37,467::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:40:37,469::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:40:37,874::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:40:37,881::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:40:37,883::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:40:39,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:40:40,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168585 s, 243 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:40:40,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:40:40,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008477 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:40:50,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:40:50,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884971 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:40:50,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:40:50,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126114 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:41:00,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:41:00,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:41:00,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103916 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:41:00,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00201093 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:41:10,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:41:10,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000717962 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:41:10,436::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 20:41:10,437::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 20:41:10,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:41:10,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00128858 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:41:10,481::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00128858 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 20:41:10,481::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 20:41:10,482::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 20:41:10,484::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 20:41:10,485::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:41:10,485::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:41:10,486::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:41:10,486::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:41:10,487::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:41:10,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:41:10,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:41:10,913::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:41:10,914::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 20:41:10,917::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 20:41:10,918::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 20:41:10,918::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 20:41:10,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:41:11,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:41:11,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:41:11,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013508 s, 303 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:41:20,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:41:20,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806112 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:41:21,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:41:21,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159591 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:41:30,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:41:30,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131913 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:41:31,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:41:31,561::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0280657 s, 146 kB/s\n'; <rc> = 0
>Thread-14590::WARNING::2013-03-17 20:41:37,911::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:41:37,917::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:41:40,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:41:40,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00068649 s, 6.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:41:41,603::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:41:41,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103522 s, 4.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:41:47,919::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:41:47,926::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:41:48,346::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:41:48,353::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:41:48,354::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:41:50,556::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:41:50,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803257 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:41:51,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:41:51,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00704182 s, 582 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:42:00,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:42:00,627::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863237 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:42:01,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:42:01,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155541 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:42:10,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:42:10,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142306 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:42:11,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:42:11,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077377 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:42:20,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:42:20,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140905 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:42:21,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:42:21,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137602 s, 298 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:42:30,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:42:30,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000933446 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:42:31,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:42:32,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150745 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:42:40,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:42:40,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808916 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:42:42,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:42:42,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00182037 s, 2.3 MB/s\n'; <rc> = 0
>Thread-14591::WARNING::2013-03-17 20:42:48,383::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:42:48,389::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:42:50,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:42:50,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141123 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:42:52,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:42:52,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807826 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:42:58,391::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 20:42:58,392::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:42:58,393::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:42:58,797::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:42:58,803::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:42:58,805::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:43:00,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:43:00,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101471 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:43:02,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:43:02,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018014 s, 227 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:43:10,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:43:10,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100285 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:43:12,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:43:12,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143922 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:43:20,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:43:20,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805019 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:43:22,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:43:22,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204916 s, 200 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:43:30,973::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 20:43:30,975::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 20:43:30,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:43:31,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00138945 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:43:31,017::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00138945 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 20:43:31,018::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 20:43:31,018::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 20:43:31,019::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 20:43:31,020::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:43:31,021::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:43:31,021::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:43:31,022::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:43:31,022::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:43:31,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:43:31,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:43:31,445::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:43:31,446::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 20:43:31,448::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 20:43:31,450::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 20:43:31,450::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 20:43:31,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:43:31,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:43:31,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:43:31,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182872 s, 224 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:43:32,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:43:32,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137984 s, 297 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:43:41,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:43:41,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016572 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:43:42,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:43:42,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176143 s, 233 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:43:51,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:43:51,958::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122805 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:43:52,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:43:52,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158936 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14592::WARNING::2013-03-17 20:43:58,832::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:43:58,843::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:44:01,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:44:02,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140741 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:44:02,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:44:02,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013875 s, 295 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:44:08,847::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:44:08,849::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:44:09,258::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:44:09,265::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:44:09,266::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:44:12,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:44:12,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897949 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:44:12,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:44:12,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.021833 s, 188 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:44:22,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:44:22,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818284 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:44:22,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:44:23,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00307193 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:44:32,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:44:32,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154286 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:44:33,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:44:33,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157699 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:44:42,129::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:44:42,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010284 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:44:43,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:44:43,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146892 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:44:52,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:44:52,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155042 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:44:53,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:44:53,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00200835 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:45:02,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:45:02,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115505 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:45:03,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:45:03,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015116 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14593::WARNING::2013-03-17 20:45:09,296::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:45:09,301::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:45:12,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:45:12,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814522 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:45:13,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:45:13,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916774 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:45:19,304::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:45:19,305::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:45:19,736::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:45:19,743::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:45:19,744::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:45:22,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:45:22,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015337 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:45:23,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:45:23,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176788 s, 232 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:45:32,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:45:32,364::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120749 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:45:33,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:45:33,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872146 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:45:42,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:45:42,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133155 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:45:43,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:45:43,658::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136379 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:45:52,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:45:52,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796893 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:45:53,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:45:53,738::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139723 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:46:02,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:46:02,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138522 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:46:03,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:46:03,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000729771 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:46:12,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:46:12,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134669 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:46:13,851::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 20:46:13,852::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 20:46:13,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:46:13,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00132021 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:46:13,897::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00132021 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 20:46:13,897::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 20:46:13,897::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 20:46:13,899::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 20:46:13,900::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:46:13,901::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:46:13,901::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:46:13,902::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:46:13,902::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:46:13,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:46:14,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:46:14,355::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:46:14,356::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 20:46:14,359::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 20:46:14,360::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 20:46:14,361::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 20:46:14,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:46:14,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:46:14,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:46:14,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00290163 s, 1.4 MB/s\n'; <rc> = 0
>Thread-14594::WARNING::2013-03-17 20:46:19,771::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:46:19,775::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:46:22,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:46:22,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795225 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:46:24,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:46:24,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00628204 s, 652 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:46:29,777::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:46:29,778::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:46:30,184::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:46:30,191::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:46:30,192::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:46:32,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:46:32,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873122 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:46:34,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:46:34,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159291 s, 257 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:46:42,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:46:42,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865929 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:46:44,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:46:45,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148072 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:46:52,647::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:46:52,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146572 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:46:55,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:46:55,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755476 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:47:02,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:47:02,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136797 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:47:05,117::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:47:05,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887236 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:47:12,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:47:12,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125727 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:47:15,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:47:15,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083798 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:47:22,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:47:22,837::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165493 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:47:25,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:47:25,299::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.009755 s, 420 kB/s\n'; <rc> = 0
>Thread-14595::WARNING::2013-03-17 20:47:30,224::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:47:30,228::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:47:32,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:47:32,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136596 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:47:35,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:47:35,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000983196 s, 4.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:47:40,234::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:47:40,235::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:47:40,630::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:47:40,637::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:47:40,638::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:47:42,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:47:42,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140288 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:47:45,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:47:45,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118836 s, 345 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:47:52,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:47:52,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813718 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:47:55,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:47:55,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074783 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:48:02,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:48:02,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161641 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:48:05,592::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:48:05,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00225715 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:48:13,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:48:13,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000715259 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:48:15,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:48:15,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135315 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:48:23,034::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:48:23,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179409 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:48:25,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:48:25,791::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00264034 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:48:33,080::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 20:48:33,084::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 20:48:33,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:48:33,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000762602 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:48:33,122::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000762602 s, 2.7 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 20:48:33,122::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 20:48:33,122::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 20:48:33,124::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 20:48:33,125::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:48:33,125::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:48:33,126::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:48:33,126::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:48:33,126::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:48:33,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:48:33,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:48:33,540::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:48:33,541::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 20:48:33,543::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 20:48:33,544::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 20:48:33,545::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 20:48:33,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:48:33,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:48:33,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:48:33,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183867 s, 223 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:48:35,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:48:35,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0091913 s, 446 kB/s\n'; <rc> = 0
>Thread-14596::WARNING::2013-03-17 20:48:40,666::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:48:40,670::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:48:43,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:48:43,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804853 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:48:45,922::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:48:45,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818667 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:48:50,672::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 20:48:50,673::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:48:50,674::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:48:51,080::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:48:51,087::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:48:51,088::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:48:53,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:48:54,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0019486 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:48:55,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:48:56,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167634 s, 244 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:49:04,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:49:04,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148746 s, 275 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:49:06,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:49:06,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886557 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:49:14,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:49:14,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136834 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:49:16,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:49:16,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148304 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:49:24,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:49:24,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112133 s, 365 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:49:26,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:49:26,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014552 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:49:34,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:49:34,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131546 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:49:36,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:49:36,364::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746529 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:49:44,260::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:49:44,299::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163498 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:49:46,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:49:46,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000947568 s, 4.3 MB/s\n'; <rc> = 0
>Thread-14597::WARNING::2013-03-17 20:49:51,117::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:49:51,121::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:49:54,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:49:54,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896919 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:49:56,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:49:56,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803405 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:50:01,123::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:50:01,124::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:50:01,804::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:50:01,811::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:50:01,813::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:50:04,345::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:50:04,401::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146283 s, 280 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:50:06,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:50:06,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163027 s, 251 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:50:14,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:50:14,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849272 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:50:16,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:50:16,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887982 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:50:24,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:50:24,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161306 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:50:26,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:50:26,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116748 s, 351 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:50:34,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:50:34,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128863 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:50:36,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:50:36,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848915 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:50:44,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:50:44,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104097 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:50:46,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:50:46,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135068 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:50:54,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:50:54,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788025 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:50:56,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:50:57,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00221536 s, 1.8 MB/s\n'; <rc> = 0
>Thread-14598::WARNING::2013-03-17 20:51:01,842::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:51:01,849::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:51:04,614::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:51:04,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013078 s, 313 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:51:07,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:51:07,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852088 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:51:11,852::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:51:11,853::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:51:12,306::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:51:12,316::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:51:12,317::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:51:14,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:51:14,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001431 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:51:17,118::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 20:51:17,120::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 20:51:17,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:51:17,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0186308 s, 110 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:51:17,169::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0186308 s, 110 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 20:51:17,170::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 20:51:17,170::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 20:51:17,172::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 20:51:17,173::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:51:17,174::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:51:17,175::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:51:17,175::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:51:17,175::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:51:17,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:51:17,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:51:17,615::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:51:17,616::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 20:51:17,618::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 20:51:17,620::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 20:51:17,620::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 20:51:17,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:51:18,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:51:18,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:51:18,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0035134 s, 1.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:51:24,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:51:24,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751927 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:51:28,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:51:28,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806675 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:51:34,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:51:34,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000730213 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:51:38,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:51:38,195::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170211 s, 241 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:51:44,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:51:44,818::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810337 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:51:48,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:51:48,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831275 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:51:54,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:51:54,866::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014397 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:51:58,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:51:58,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158309 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:52:04,871::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:52:04,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127822 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:52:08,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:52:08,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156441 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14599::WARNING::2013-03-17 20:52:12,347::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:52:12,353::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:52:14,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:52:14,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772636 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:52:18,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:52:18,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802924 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:52:22,355::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:52:22,357::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:52:22,763::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:52:22,770::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:52:22,771::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:52:24,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:52:24,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754314 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:52:28,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:52:28,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00984853 s, 416 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:52:34,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:52:35,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134383 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:52:38,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:52:38,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841144 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:52:45,033::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:52:45,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161755 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:52:48,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:52:48,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904387 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:52:55,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:52:55,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00209114 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:52:58,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:52:58,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000910004 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:53:05,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:53:05,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000716621 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:53:08,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:53:08,899::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000717318 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:53:15,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:53:15,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154081 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:53:18,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:53:18,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170951 s, 2.4 MB/s\n'; <rc> = 0
>Thread-14600::WARNING::2013-03-17 20:53:22,808::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:53:22,811::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:53:25,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:53:25,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077498 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:53:29,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:53:29,054::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920694 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:53:32,814::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:53:32,815::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:53:33,228::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:53:33,235::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:53:33,237::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:53:35,248::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 20:53:35,251::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 20:53:35,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:53:35,287::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00131405 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:53:35,288::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00131405 s, 1.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 20:53:35,289::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 20:53:35,289::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 20:53:35,291::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 20:53:35,292::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:53:35,292::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:53:35,293::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:53:35,293::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:53:35,293::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:53:35,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:53:35,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:53:35,738::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:53:35,738::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 20:53:35,741::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 20:53:35,742::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 20:53:35,742::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 20:53:35,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:53:36,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:53:36,108::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:53:36,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152915 s, 268 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:53:39,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:53:39,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00935516 s, 438 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:53:46,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:53:46,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160414 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:53:49,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:53:49,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784267 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:53:56,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:53:56,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162526 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:53:59,254::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:53:59,299::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00197031 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:54:06,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:54:06,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084241 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:54:09,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:54:09,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014916 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:54:16,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:54:16,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764873 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:54:19,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:54:19,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861779 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:54:26,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:54:26,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147106 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:54:29,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:54:29,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167494 s, 2.4 MB/s\n'; <rc> = 0
>Thread-14601::WARNING::2013-03-17 20:54:33,264::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:54:33,271::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:54:36,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:54:36,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873445 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:54:39,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:54:39,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122043 s, 3.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:54:43,273::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 20:54:43,274::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:54:43,275::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:54:43,683::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:54:43,691::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:54:43,692::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:54:46,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:54:46,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789839 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:54:49,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:54:49,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111286 s, 368 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:54:56,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:54:56,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015257 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:54:59,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:54:59,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162961 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:55:06,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:55:06,552::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132991 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:55:09,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:55:09,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174221 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:55:16,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:55:16,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110737 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:55:19,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:55:19,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770416 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:55:26,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:55:26,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119118 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:55:29,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:55:29,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783135 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:55:36,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:55:36,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747197 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:55:40,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:55:40,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000937972 s, 4.4 MB/s\n'; <rc> = 0
>Thread-14602::WARNING::2013-03-17 20:55:43,720::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:55:43,723::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:55:46,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:55:46,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807685 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:55:50,117::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:55:50,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104443 s, 3.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:55:53,728::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:55:53,729::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:55:54,164::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:55:54,171::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:55:54,172::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:55:56,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:55:56,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848751 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:56:00,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:56:00,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108907 s, 376 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:56:06,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:56:06,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00187738 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:56:10,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:56:10,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850645 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:56:16,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:56:16,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00526495 s, 778 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:56:20,341::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 20:56:20,343::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 20:56:20,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:56:20,382::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00123797 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:56:20,383::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00123797 s, 1.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 20:56:20,384::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 20:56:20,384::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 20:56:20,385::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 20:56:20,386::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:56:20,387::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:56:20,388::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:56:20,388::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:56:20,388::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 20:56:20,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:56:20,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:56:20,843::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 20:56:20,845::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 20:56:20,848::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 20:56:20,849::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 20:56:20,849::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 20:56:20,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:56:21,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:56:21,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:56:21,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163596 s, 250 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:56:26,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:56:26,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849728 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:56:31,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:56:31,408::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014265 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:56:36,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:56:36,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849601 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:56:41,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:56:41,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167442 s, 245 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:56:46,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:56:46,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139083 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:56:51,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:56:51,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801023 s, 5.1 MB/s\n'; <rc> = 0
>Thread-14603::WARNING::2013-03-17 20:56:54,201::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:56:54,207::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:56:56,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:56:57,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144147 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:57:01,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:57:01,657::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826107 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:57:04,209::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:57:04,210::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:57:04,632::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:57:04,639::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:57:04,640::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:57:07,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:57:07,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890303 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:57:11,697::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:57:11,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189935 s, 216 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:57:17,045::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:57:17,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789102 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:57:21,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:57:21,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089769 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:57:27,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:57:27,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824422 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:57:31,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:57:31,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0213347 s, 192 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:57:37,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:57:37,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781818 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:57:41,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:57:42,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108576 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:57:47,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:57:47,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136486 s, 300 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:57:52,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:57:52,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100753 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:57:57,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:57:57,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145392 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:58:02,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:58:02,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000935748 s, 4.4 MB/s\n'; <rc> = 0
>Thread-14604::WARNING::2013-03-17 20:58:04,668::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:58:04,671::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:58:07,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:58:07,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00836963 s, 489 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:58:12,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:58:12,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153321 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:58:14,673::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:58:14,700::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:58:15,158::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:58:15,167::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:58:15,168::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:58:17,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:58:17,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852181 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:58:22,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:58:22,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178812 s, 229 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:58:27,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:58:27,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777324 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:58:32,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:58:32,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178519 s, 229 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:58:37,365::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 20:58:37,366::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 20:58:37,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:58:37,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00131107 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:58:37,411::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00131107 s, 1.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 20:58:37,412::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 20:58:37,412::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 20:58:37,414::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 20:58:37,415::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:58:37,415::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:58:37,416::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:58:37,416::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:58:37,416::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 20:58:37,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:58:37,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:58:37,856::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:58:37,857::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 20:58:37,859::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 20:58:37,860::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 20:58:37,861::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 20:58:37,863::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:58:38,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:58:38,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:58:38,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00286563 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:58:42,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:58:42,517::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.020955 s, 195 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:58:48,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:58:48,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081139 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:58:52,560::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:58:52,598::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812873 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:58:58,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:58:58,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128219 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:59:02,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:59:02,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172158 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:59:08,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:59:08,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00303129 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:59:12,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:59:12,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011337 s, 3.6 MB/s\n'; <rc> = 0
>Thread-14605::WARNING::2013-03-17 20:59:15,205::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 20:59:15,211::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 20:59:18,395::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:59:18,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163017 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:59:22,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:59:22,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857476 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 20:59:25,214::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 20:59:25,216::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 20:59:25,732::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 20:59:25,739::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 20:59:25,741::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 20:59:28,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:59:28,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900052 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:59:32,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:59:32,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177249 s, 231 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:59:38,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:59:38,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144554 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:59:42,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:59:42,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081157 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:59:48,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:59:48,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141035 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 20:59:53,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 20:59:53,046::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824658 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 20:59:58,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 20:59:58,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000662006 s, 6.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:00:03,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:00:03,130::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162638 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:00:08,614::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:00:08,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164679 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:00:13,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:00:13,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107358 s, 382 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:00:18,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:00:18,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111068 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:00:23,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:00:23,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000979828 s, 4.2 MB/s\n'; <rc> = 0
>Thread-14606::WARNING::2013-03-17 21:00:25,766::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:00:25,772::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 21:00:28,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:00:28,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789384 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:00:33,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:00:33,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788188 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:00:35,810::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 21:00:35,811::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:00:35,812::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:00:36,244::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:00:36,250::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:00:36,252::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:00:38,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:00:38,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010378 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:00:43,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:00:43,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018667 s, 219 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:00:48,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:00:48,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013402 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:00:53,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:00:53,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140098 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:00:58,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:00:58,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00184608 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:01:03,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:01:03,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0207559 s, 197 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:01:08,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:01:08,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135618 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:01:13,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:01:13,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740521 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:01:18,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:01:18,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134845 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:01:23,738::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 21:01:23,739::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 21:01:23,740::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:01:23,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00125094 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:01:23,771::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00125094 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 21:01:23,771::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 21:01:23,771::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 21:01:23,773::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 21:01:23,774::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:01:23,775::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:01:23,775::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:01:23,776::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:01:23,776::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:01:23,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:01:24,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:01:24,218::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:01:24,219::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 21:01:24,223::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 21:01:24,224::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 21:01:24,224::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 21:01:24,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:01:24,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:01:24,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:01:24,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00812297 s, 504 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:01:28,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:01:28,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153723 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:01:34,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:01:34,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000727013 s, 5.6 MB/s\n'; <rc> = 0
>Thread-14607::WARNING::2013-03-17 21:01:36,280::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:01:36,284::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 21:01:38,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:01:39,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102026 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:01:45,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:01:45,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862569 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:01:46,286::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:01:46,288::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:01:46,726::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:01:46,733::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:01:46,734::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:01:49,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:01:49,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901967 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:01:55,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:01:55,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01405 s, 292 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:01:59,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:01:59,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839408 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:02:05,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:02:05,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792063 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:02:09,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:02:09,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00195032 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:02:15,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:02:15,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162604 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:02:19,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:02:19,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801029 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:02:25,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:02:25,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849534 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:02:29,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:02:29,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153452 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:02:35,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:02:35,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203712 s, 201 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:02:39,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:02:39,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151709 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:02:45,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:02:45,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774073 s, 5.3 MB/s\n'; <rc> = 0
>Thread-14608::WARNING::2013-03-17 21:02:46,760::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:02:46,764::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 21:02:49,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:02:49,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787592 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:02:55,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:02:55,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775132 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:02:56,767::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:02:56,768::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:02:57,202::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:02:57,209::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:02:57,210::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:02:59,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:02:59,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851494 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:03:06,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:03:06,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102031 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:03:09,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:03:09,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129852 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:03:16,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:03:16,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785085 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:03:19,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:03:19,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126361 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:03:26,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:03:26,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100597 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:03:29,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:03:29,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892192 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:03:36,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:03:36,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780143 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:03:39,418::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 21:03:39,420::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 21:03:39,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:03:39,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00159393 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:03:39,468::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00159393 s, 1.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 21:03:39,469::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 21:03:39,469::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 21:03:39,470::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 21:03:39,471::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:03:39,472::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:03:39,473::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:03:39,473::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:03:39,473::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:03:39,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:03:39,880::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:03:39,884::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:03:39,885::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 21:03:39,890::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 21:03:39,893::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 21:03:39,893::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 21:03:39,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:03:40,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:03:40,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:03:40,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135232 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:03:46,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:03:46,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135847 s, 302 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:03:50,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:03:50,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793643 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:03:56,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:03:56,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00218978 s, 1.9 MB/s\n'; <rc> = 0
>Thread-14609::WARNING::2013-03-17 21:03:57,247::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:03:57,251::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 21:04:00,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:04:00,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136624 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:04:06,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:04:06,777::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145841 s, 281 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:04:07,253::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:04:07,255::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:04:07,683::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:04:07,692::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:04:07,693::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:04:10,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:04:10,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016544 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:04:16,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:04:16,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795549 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:04:20,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:04:20,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855393 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:04:26,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:04:26,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860048 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:04:30,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:04:30,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000922806 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:04:36,959::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:04:36,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819942 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:04:40,492::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:04:40,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000986751 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:04:47,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:04:47,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177899 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:04:50,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:04:50,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139272 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:04:57,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:04:57,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152239 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:05:00,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:05:00,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156362 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:05:07,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:05:07,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00601178 s, 681 kB/s\n'; <rc> = 0
>Thread-14610::WARNING::2013-03-17 21:05:07,722::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:05:07,728::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 21:05:10,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:05:10,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165396 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:05:17,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:05:17,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146652 s, 279 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:05:17,731::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:05:17,733::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:05:18,147::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:05:18,154::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:05:18,155::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:05:20,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:05:20,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087167 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:05:27,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:05:27,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842105 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:05:30,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:05:30,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00415694 s, 985 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:05:37,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:05:37,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788092 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:05:40,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:05:40,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139448 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:05:47,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:05:47,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775234 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:05:50,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:05:50,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00398727 s, 1.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:05:57,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:05:57,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141369 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:06:00,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:06:00,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154009 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:06:07,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:06:07,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756936 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:06:10,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:06:10,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893019 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:06:17,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:06:18,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141279 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14611::WARNING::2013-03-17 21:06:18,188::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:06:18,200::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 21:06:20,922::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:06:20,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880819 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:06:28,214::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 21:06:28,215::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:06:28,216::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:06:28,238::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 21:06:28,243::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 21:06:28,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:06:28,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00854084 s, 240 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:06:28,305::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00854084 s, 240 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 21:06:28,305::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 21:06:28,305::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 21:06:28,307::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 21:06:28,308::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm reload operation' is holding the operation mutex, waiting...
>Thread-21::DEBUG::2013-03-17 21:06:28,620::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:06:28,626::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:06:28,626::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:06:28,658::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:06:28,666::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:06:28,667::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:06:28,668::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:06:28,669::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:06:28,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:06:29,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:06:29,062::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:06:29,063::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 21:06:29,066::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 21:06:29,067::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 21:06:29,067::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 21:06:29,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:06:29,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:06:29,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:06:29,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138054 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:06:30,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:06:31,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754408 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:06:39,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:06:39,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00492226 s, 832 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:06:41,014::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:06:41,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845664 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:06:49,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:06:49,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141104 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:06:51,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:06:51,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757359 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:06:59,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:06:59,681::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836384 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:07:01,092::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:07:01,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147414 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:07:09,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:07:09,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771762 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:07:11,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:07:11,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000950811 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:07:19,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:07:19,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147214 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:07:21,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:07:21,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104488 s, 3.9 MB/s\n'; <rc> = 0
>Thread-14612::WARNING::2013-03-17 21:07:28,677::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:07:28,683::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 21:07:29,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:07:29,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086045 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:07:31,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:07:31,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115171 s, 356 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:07:38,686::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:07:38,688::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:07:39,109::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:07:39,116::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:07:39,117::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:07:39,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:07:40,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101925 s, 402 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:07:41,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:07:41,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172738 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:07:50,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:07:50,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843432 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:07:51,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:07:51,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834054 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:08:00,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:08:00,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151772 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:08:01,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:08:01,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00631553 s, 649 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:08:10,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:08:10,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772661 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:08:11,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:08:11,617::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000922653 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:08:20,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:08:20,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0157892 s, 259 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:08:21,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:08:21,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000730201 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:08:30,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:08:30,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134546 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:08:31,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:08:31,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794998 s, 5.2 MB/s\n'; <rc> = 0
>Thread-14613::WARNING::2013-03-17 21:08:39,157::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:08:39,162::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 21:08:40,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:08:40,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151556 s, 270 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:08:41,737::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 21:08:41,738::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 21:08:41,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:08:41,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000799605 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:08:41,769::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000799605 s, 2.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 21:08:41,770::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 21:08:41,770::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 21:08:41,772::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 21:08:41,773::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:08:41,773::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:08:41,774::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:08:41,774::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:08:41,775::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:08:41,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:08:42,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:08:42,208::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:08:42,209::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 21:08:42,211::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 21:08:42,213::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 21:08:42,213::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 21:08:42,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:08:42,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:08:42,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:08:42,611::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00469013 s, 873 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:08:49,165::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:08:49,166::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:08:49,586::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:08:49,593::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:08:49,594::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:08:50,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:08:50,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127049 s, 322 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:08:52,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:08:52,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829065 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:09:00,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:09:00,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110851 s, 370 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:09:02,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:09:02,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787814 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:09:10,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:09:10,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138715 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:09:12,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:09:12,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178153 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:09:20,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:09:20,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000956166 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:09:22,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:09:22,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875349 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:09:30,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:09:30,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00469273 s, 873 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:09:32,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:09:32,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000704695 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:09:40,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:09:41,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146345 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:09:42,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:09:42,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148691 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14614::WARNING::2013-03-17 21:09:49,639::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:09:49,644::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 21:09:51,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:09:51,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113364 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:09:52,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:09:52,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868145 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:09:59,648::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:09:59,649::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:10:00,061::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:10:00,068::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:10:00,069::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:10:01,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:10:01,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136586 s, 300 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:10:02,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:10:02,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000973606 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:10:11,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:10:11,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165613 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:10:12,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:10:12,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839522 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:10:21,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:10:21,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158213 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:10:23,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:10:23,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749193 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:10:31,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:10:31,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163831 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:10:33,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:10:33,081::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170811 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:10:41,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:10:41,448::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867504 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:10:43,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:10:43,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140536 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:10:51,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:10:51,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105588 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:10:53,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:10:53,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146649 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14615::WARNING::2013-03-17 21:11:00,097::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:11:00,105::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 21:11:01,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:11:01,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000693178 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:11:03,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:11:03,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163954 s, 2.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:11:10,111::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:11:10,113::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:11:10,513::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:11:10,520::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:11:10,521::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:11:11,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:11:11,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130975 s, 313 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:11:13,227::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:11:13,270::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160221 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:11:21,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:11:21,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790567 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:11:23,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:11:23,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015138 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:11:31,792::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 21:11:31,794::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 21:11:31,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:11:31,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000780462 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:11:31,826::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000780462 s, 2.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 21:11:31,826::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 21:11:31,827::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 21:11:31,828::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 21:11:31,829::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:11:31,830::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:11:31,830::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:11:31,831::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:11:31,831::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:11:31,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:11:32,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:11:32,259::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:11:32,260::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 21:11:32,262::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 21:11:32,263::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 21:11:32,264::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 21:11:32,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:11:32,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:11:32,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:11:32,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00717812 s, 571 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:11:33,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:11:33,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180818 s, 227 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:11:42,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:11:42,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00236674 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:11:43,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:11:43,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146901 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:11:52,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:11:52,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140821 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:11:53,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:11:53,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761277 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:12:02,910::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:12:02,936::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865977 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:12:03,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:12:03,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00338016 s, 1.2 MB/s\n'; <rc> = 0
>Thread-14616::WARNING::2013-03-17 21:12:10,546::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:12:10,550::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 21:12:12,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:12:13,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083489 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:12:13,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:12:13,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101066 s, 4.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:12:20,551::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 21:12:20,553::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:12:20,554::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:12:20,953::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:12:20,960::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:12:20,961::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:12:23,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:12:23,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138937 s, 295 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:12:23,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:12:23,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871767 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:12:33,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:12:33,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0271756 s, 151 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:12:33,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:12:33,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0226581 s, 181 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:12:43,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:12:43,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142553 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:12:43,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:12:43,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00201105 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:12:53,445::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:12:53,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145138 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:12:53,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:12:53,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145548 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:13:03,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:13:03,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149846 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:13:03,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:13:03,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788569 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:13:13,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:13:13,636::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000730359 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:13:13,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:13:13,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013919 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14617::WARNING::2013-03-17 21:13:20,988::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:13:20,992::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 21:13:23,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:13:23,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880194 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:13:23,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:13:23,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111626 s, 3.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:13:30,994::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:13:30,996::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:13:31,413::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:13:31,420::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:13:31,421::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:13:33,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:13:33,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192877 s, 212 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:13:33,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:13:33,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890587 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:13:43,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:13:43,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135954 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:13:43,947::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 21:13:43,949::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 21:13:43,950::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:13:43,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00121579 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:13:43,988::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00121579 s, 1.7 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 21:13:43,989::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 21:13:43,989::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 21:13:43,990::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 21:13:43,991::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:13:43,992::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:13:43,992::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:13:43,993::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:13:43,993::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:13:43,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:13:44,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:13:44,440::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:13:44,441::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 21:13:44,443::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 21:13:44,445::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 21:13:44,445::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 21:13:44,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:13:44,853::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:13:44,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:13:44,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013634 s, 300 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:13:53,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:13:53,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135149 s, 303 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:13:54,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:13:54,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083166 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:14:04,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:14:04,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0096125 s, 426 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:14:04,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:14:05,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014346 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:14:14,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:14:14,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134019 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:14:15,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:14:15,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136699 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:14:24,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:14:24,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008318 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:14:25,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:14:25,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860102 s, 4.8 MB/s\n'; <rc> = 0
>Thread-14618::WARNING::2013-03-17 21:14:31,452::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:14:31,458::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 21:14:34,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:14:34,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871861 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:14:35,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:14:35,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00073852 s, 5.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:14:41,463::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:14:41,464::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:14:41,937::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:14:41,944::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:14:41,945::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:14:44,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:14:44,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170565 s, 240 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:14:45,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:14:45,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146425 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:14:54,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:14:54,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000953649 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:14:55,281::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:14:55,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00220382 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:15:04,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:15:04,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000982682 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:15:05,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:15:05,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164956 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:15:14,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:15:14,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149505 s, 274 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:15:15,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:15:15,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129781 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:15:24,648::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:15:24,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107662 s, 380 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:15:25,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:15:25,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125333 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:15:34,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:15:34,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0200628 s, 204 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:15:35,606::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:15:35,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802534 s, 5.1 MB/s\n'; <rc> = 0
>Thread-14619::WARNING::2013-03-17 21:15:41,975::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:15:41,982::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 21:15:44,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:15:44,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849851 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:15:45,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:15:45,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853993 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:15:51,985::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:15:51,986::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:15:52,423::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:15:52,429::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:15:52,431::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:15:54,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:15:54,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141126 s, 290 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:15:55,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:15:55,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0036658 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:16:04,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:16:05,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839407 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:16:05,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:16:05,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101302 s, 404 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:16:15,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:16:15,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108553 s, 377 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:16:15,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:16:15,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876419 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:16:25,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:16:25,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878986 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:16:26,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:16:26,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140493 s, 292 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:16:35,213::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 21:16:35,214::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 21:16:35,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:16:35,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0144446 s, 142 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:16:35,266::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0144446 s, 142 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 21:16:35,267::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 21:16:35,267::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 21:16:35,269::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 21:16:35,271::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:16:35,272::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:16:35,272::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:16:35,273::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:16:35,273::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:16:35,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:16:35,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:16:35,705::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:16:35,706::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 21:16:35,709::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 21:16:35,710::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 21:16:35,710::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 21:16:35,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:16:36,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:16:36,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130858 s, 313 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:16:36,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:16:36,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:16:36,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160662 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:16:46,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:16:46,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000742326 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:16:46,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:16:46,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015061 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14620::WARNING::2013-03-17 21:16:52,466::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:16:52,471::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 21:16:56,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:16:56,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796998 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:16:56,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:16:56,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837546 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:17:02,479::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:17:02,480::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:17:02,872::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:17:02,879::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:17:02,880::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:17:06,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:17:06,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00347735 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:17:06,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:17:06,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145449 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:17:16,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:17:16,399::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181464 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:17:16,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:17:16,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075893 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:17:26,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:17:26,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144585 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:17:26,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:17:26,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108396 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:17:36,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:17:36,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834455 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:17:36,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:17:36,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000910959 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:17:46,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:17:46,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155203 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:17:46,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:17:46,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080761 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:17:56,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:17:56,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151879 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:17:56,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:17:56,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00947001 s, 433 kB/s\n'; <rc> = 0
>Thread-14621::WARNING::2013-03-17 21:18:02,913::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:18:02,916::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 21:18:06,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:18:06,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123163 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:18:07,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:18:07,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848458 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:18:12,923::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 21:18:12,924::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:18:12,924::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:18:13,351::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:18:13,357::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:18:13,359::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:18:16,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:18:16,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078445 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:18:17,079::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:18:17,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00847701 s, 483 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:18:26,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:18:26,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875657 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:18:27,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:18:27,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148938 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:18:36,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:18:36,733::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015981 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:18:37,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:18:37,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786603 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:18:46,738::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 21:18:46,739::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 21:18:46,741::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:18:46,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00152685 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:18:46,779::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00152685 s, 1.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 21:18:46,780::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 21:18:46,780::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 21:18:46,782::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 21:18:46,783::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:18:46,783::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:18:46,784::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:18:46,784::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:18:46,784::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:18:46,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:18:47,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:18:47,212::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:18:47,213::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 21:18:47,219::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 21:18:47,220::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 21:18:47,220::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 21:18:47,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:18:47,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:18:47,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147212 s, 278 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:18:47,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:18:47,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:18:47,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123912 s, 331 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:18:57,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:18:57,494::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113831 s, 360 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:18:57,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:18:57,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158473 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:19:07,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:19:07,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141409 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:19:07,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:19:07,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000689538 s, 5.9 MB/s\n'; <rc> = 0
>Thread-14622::WARNING::2013-03-17 21:19:13,385::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:19:13,390::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 21:19:17,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:19:17,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111774 s, 366 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:19:17,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:19:17,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151381 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:19:23,392::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:19:23,394::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:19:23,869::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:19:23,876::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:19:23,877::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:19:27,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:19:27,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171005 s, 240 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:19:27,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:19:27,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156786 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:19:37,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:19:37,853::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152365 s, 269 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:19:37,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:19:37,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00188655 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:19:47,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:19:48,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:19:48,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133505 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:19:48,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087539 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:19:58,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:19:58,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887768 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:19:58,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:19:58,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141426 s, 290 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:20:08,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:20:08,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858009 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:20:08,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:20:08,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148423 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:20:18,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:20:18,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831776 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:20:18,281::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:20:18,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141911 s, 289 kB/s\n'; <rc> = 0
>Thread-14623::WARNING::2013-03-17 21:20:23,905::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:20:23,908::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 21:20:28,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:20:28,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833664 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:20:28,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:20:28,402::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801212 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:20:33,913::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:20:33,914::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:20:34,375::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:20:34,381::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:20:34,383::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:20:38,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:20:38,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085309 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:20:38,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:20:38,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171432 s, 239 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:20:48,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:20:48,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140636 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:20:48,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:20:48,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843291 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:20:58,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:20:58,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080265 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:20:58,605::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:20:58,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831578 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:21:08,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:21:08,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895129 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:21:08,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:21:08,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122748 s, 334 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:21:18,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:21:18,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133721 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:21:18,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:21:18,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00799801 s, 512 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:21:28,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:21:28,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141146 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:21:28,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:21:28,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00253363 s, 1.6 MB/s\n'; <rc> = 0
>Thread-14624::WARNING::2013-03-17 21:21:34,412::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:21:34,416::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 21:21:38,471::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:21:38,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856581 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:21:38,930::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 21:21:38,932::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 21:21:38,933::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:21:38,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000808852 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:21:38,959::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000808852 s, 2.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 21:21:38,959::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 21:21:38,959::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 21:21:38,961::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 21:21:38,962::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:21:38,963::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:21:38,963::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:21:38,964::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:21:38,964::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:21:38,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:21:39,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:21:39,401::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:21:39,402::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 21:21:39,405::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 21:21:39,406::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 21:21:39,407::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 21:21:39,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:21:39,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:21:39,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:21:39,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00600738 s, 682 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:21:44,420::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:21:44,421::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:21:44,877::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:21:44,883::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:21:44,885::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:21:48,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:21:48,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805128 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:21:49,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:21:49,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873853 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:21:58,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:21:58,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884865 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:21:59,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:21:59,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839289 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:22:08,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:22:08,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141682 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:22:09,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:22:10,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145279 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:22:18,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:22:18,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017986 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:22:20,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:22:20,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163433 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:22:28,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:22:28,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163056 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:22:30,128::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:22:30,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151459 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:22:38,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:22:38,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830549 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:22:40,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:22:40,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163067 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14625::WARNING::2013-03-17 21:22:44,909::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:22:44,913::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 21:22:48,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:22:48,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00455539 s, 899 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:22:50,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:22:50,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197715 s, 207 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:22:54,922::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:22:54,923::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:22:55,331::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:22:55,337::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:22:55,339::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:22:58,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:22:58,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172079 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:23:00,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:23:00,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156718 s, 261 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:23:08,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:23:08,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815102 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:23:10,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:23:10,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164374 s, 249 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:23:18,861::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:23:18,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130628 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:23:20,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:23:20,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174314 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:23:28,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:23:28,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131735 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:23:30,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:23:30,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011277 s, 363 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:23:38,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:23:39,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010086 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:23:40,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:23:40,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147457 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:23:49,010::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 21:23:49,011::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 21:23:49,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:23:49,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0012036 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:23:49,051::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0012036 s, 1.7 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 21:23:49,052::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 21:23:49,052::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 21:23:49,053::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 21:23:49,057::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:23:49,058::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:23:49,058::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:23:49,058::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:23:49,059::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:23:49,060::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:23:49,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:23:49,490::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:23:49,491::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 21:23:49,494::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 21:23:49,495::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 21:23:49,495::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 21:23:49,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:23:49,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:23:49,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:23:49,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00865966 s, 473 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:23:50,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:23:50,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153402 s, 267 kB/s\n'; <rc> = 0
>Thread-14626::WARNING::2013-03-17 21:23:55,365::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:23:55,369::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 21:23:59,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:23:59,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812947 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:24:00,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:24:01,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167377 s, 2.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:24:05,370::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 21:24:05,371::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:24:05,372::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:24:05,837::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:24:05,844::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:24:05,845::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:24:09,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:24:09,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749567 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:24:11,079::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:24:11,123::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191502 s, 214 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:24:20,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:24:20,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878157 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:24:21,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:24:21,188::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869705 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:24:30,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:24:30,081::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164441 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:24:31,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:24:31,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151258 s, 271 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:24:40,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:24:40,130::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00603675 s, 679 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:24:41,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:24:41,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858155 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:24:50,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:24:50,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140737 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:24:51,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:24:51,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153703 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:25:00,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:25:00,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174269 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:25:01,481::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:25:01,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00279954 s, 1.5 MB/s\n'; <rc> = 0
>Thread-14627::WARNING::2013-03-17 21:25:05,869::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:25:05,872::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 21:25:10,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:25:10,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00094131 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:25:11,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:25:11,607::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081543 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:25:15,875::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:25:15,877::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:25:16,286::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:25:16,293::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:25:16,294::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:25:20,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:25:20,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788028 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:25:21,647::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:25:21,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120675 s, 339 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:25:30,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:25:30,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849121 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:25:31,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:25:31,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00649646 s, 630 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:25:40,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:25:40,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155329 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:25:41,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:25:41,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104989 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:25:50,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:25:50,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00535192 s, 765 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:25:51,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:25:51,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119044 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:26:00,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:26:00,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793038 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:26:01,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:26:01,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155989 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:26:10,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:26:10,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149136 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:26:12,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:26:12,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143991 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14628::WARNING::2013-03-17 21:26:16,321::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:26:16,325::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 21:26:20,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:26:20,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000964868 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:26:22,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:26:22,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087302 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:26:26,327::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:26:26,328::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:26:26,743::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:26:26,749::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:26:26,751::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:26:30,532::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:26:30,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743695 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:26:32,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:26:32,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0239242 s, 171 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:26:40,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:26:40,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886981 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:26:42,329::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 21:26:42,331::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 21:26:42,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:26:42,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00256991 s, 797 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:26:42,393::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00256991 s, 797 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 21:26:42,393::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 21:26:42,394::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 21:26:42,407::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 21:26:42,408::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:26:42,409::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:26:42,409::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:26:42,410::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:26:42,410::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:26:42,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:26:42,820::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:26:42,824::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:26:42,825::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 21:26:42,828::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 21:26:42,829::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 21:26:42,829::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 21:26:42,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:26:43,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:26:43,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:26:43,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0554393 s, 73.9 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:26:50,603::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:26:50,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136432 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:26:53,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:26:53,350::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150894 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:27:00,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:27:00,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00195914 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:27:03,405::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:27:03,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104159 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:27:10,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:27:10,742::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119845 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:27:13,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:27:13,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842878 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:27:20,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:27:20,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142952 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:27:23,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:27:23,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845592 s, 4.8 MB/s\n'; <rc> = 0
>Thread-14629::WARNING::2013-03-17 21:27:26,777::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:27:26,780::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 21:27:30,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:27:30,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000928597 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:27:33,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:27:33,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077217 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:27:36,782::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:27:36,784::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:27:37,202::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:27:37,209::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:27:37,210::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:27:40,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:27:40,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168385 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:27:43,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:27:43,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891098 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:27:50,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:27:50,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142301 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:27:53,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:27:53,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143558 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:28:00,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:28:00,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826327 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:28:03,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:28:03,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145868 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:28:10,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:28:11,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138871 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:28:13,950::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:28:13,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174386 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:28:21,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:28:21,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145722 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:28:24,021::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:28:24,060::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011731 s, 349 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:28:31,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:28:31,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145273 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:28:34,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:28:34,140::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00206856 s, 2.0 MB/s\n'; <rc> = 0
>Thread-14630::WARNING::2013-03-17 21:28:37,238::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:28:37,242::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 21:28:41,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:28:41,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150042 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:28:44,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:28:44,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139755 s, 293 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:28:47,244::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:28:47,246::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:28:47,696::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:28:47,703::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:28:47,705::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:28:51,150::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 21:28:51,153::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 21:28:51,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:28:51,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000781624 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:28:51,189::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000781624 s, 2.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 21:28:51,189::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 21:28:51,190::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 21:28:51,192::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 21:28:51,193::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:28:51,194::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:28:51,194::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:28:51,195::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:28:51,195::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:28:51,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:28:51,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:28:51,645::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:28:51,646::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 21:28:51,648::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 21:28:51,650::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 21:28:51,650::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 21:28:51,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:28:52,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:28:52,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:28:52,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129127 s, 317 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:28:54,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:28:54,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816007 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:29:02,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:29:02,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142075 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:29:04,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:29:04,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156653 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:29:12,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:29:12,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081565 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:29:14,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:29:14,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906611 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:29:22,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:29:22,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008254 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:29:24,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:29:24,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115785 s, 354 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:29:32,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:29:32,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128863 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:29:34,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:29:34,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141551 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:29:42,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:29:42,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143651 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:29:44,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:29:44,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000961183 s, 4.3 MB/s\n'; <rc> = 0
>Thread-14631::WARNING::2013-03-17 21:29:47,730::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:29:47,734::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 21:29:52,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:29:52,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820345 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:29:54,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:29:54,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784792 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:29:57,735::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 21:29:57,736::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:29:57,737::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:29:58,222::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:29:58,228::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:29:58,230::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:30:02,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:30:02,358::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165685 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:30:04,837::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:30:04,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163781 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:30:12,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:30:12,392::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890219 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:30:14,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:30:14,944::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766887 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:30:22,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:30:22,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166131 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:30:24,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:30:25,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074263 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:30:32,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:30:32,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172371 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:30:35,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:30:35,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177618 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:30:42,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:30:42,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900517 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:30:45,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:30:45,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163628 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:30:52,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:30:52,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082323 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:30:55,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:30:55,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133356 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14632::WARNING::2013-03-17 21:30:58,276::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:30:58,288::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 21:31:02,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:31:02,611::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771799 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:31:05,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:31:05,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112256 s, 3.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:31:08,291::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:31:08,292::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:31:08,758::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:31:08,765::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:31:08,767::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:31:12,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:31:12,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839941 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:31:15,396::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:31:15,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849324 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:31:22,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:31:22,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139624 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:31:25,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:31:25,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135004 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:31:32,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:31:32,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836542 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:31:35,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:31:35,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00263783 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:31:42,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:31:42,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828261 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:31:45,620::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 21:31:45,621::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 21:31:45,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:31:45,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00145262 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:31:45,670::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00145262 s, 1.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 21:31:45,672::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 21:31:45,673::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 21:31:45,674::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 21:31:45,675::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:31:45,676::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:31:45,677::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:31:45,677::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:31:45,677::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:31:45,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:31:46,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:31:46,112::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:31:46,113::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 21:31:46,116::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 21:31:46,117::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 21:31:46,117::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 21:31:46,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:31:46,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:31:46,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:31:46,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0090327 s, 453 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:31:52,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:31:52,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876154 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:31:56,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:31:56,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165563 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:32:02,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:32:02,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100012 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:32:06,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:32:06,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909136 s, 4.5 MB/s\n'; <rc> = 0
>Thread-14633::WARNING::2013-03-17 21:32:08,793::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:32:08,798::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 21:32:12,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:32:12,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812299 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:32:16,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:32:16,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794273 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:32:18,846::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:32:18,847::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:32:19,293::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:32:19,299::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:32:19,301::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:32:22,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:32:22,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790077 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:32:26,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:32:26,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079713 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:32:32,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:32:32,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151098 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:32:36,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:32:36,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151845 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:32:42,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:32:42,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778694 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:32:46,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:32:47,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148789 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:32:53,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:32:53,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775339 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:32:57,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:32:57,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173465 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:33:03,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:33:03,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136279 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:33:07,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:33:07,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014536 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:33:13,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:33:13,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108326 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:33:17,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:33:17,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000700517 s, 5.8 MB/s\n'; <rc> = 0
>Thread-14634::WARNING::2013-03-17 21:33:19,330::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:33:19,336::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 21:33:23,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:33:23,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112219 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:33:27,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:33:27,366::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865567 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:33:29,339::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:33:29,340::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:33:29,766::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:33:29,772::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:33:29,774::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:33:33,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:33:33,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169957 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:33:37,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:33:37,439::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767453 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:33:43,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:33:43,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000991882 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:33:47,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:33:47,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154928 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:33:53,248::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 21:33:53,250::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 21:33:53,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:33:53,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000812999 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:33:53,281::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000812999 s, 2.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 21:33:53,281::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 21:33:53,282::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 21:33:53,287::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 21:33:53,288::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:33:53,289::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:33:53,289::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:33:53,289::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:33:53,290::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:33:53,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:33:53,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:33:53,716::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:33:53,717::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 21:33:53,719::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 21:33:53,721::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 21:33:53,721::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 21:33:53,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:33:54,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:33:54,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:33:54,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141641 s, 289 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:33:57,560::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:33:57,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00250779 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:34:04,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:34:04,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000739404 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:34:07,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:34:07,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0009169 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:34:14,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:34:14,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00820997 s, 499 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:34:17,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:34:17,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016525 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:34:24,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:34:24,309::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120703 s, 339 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:34:27,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:34:27,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00382226 s, 1.1 MB/s\n'; <rc> = 0
>Thread-14635::WARNING::2013-03-17 21:34:29,797::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:34:29,801::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 21:34:34,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:34:34,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000959294 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:34:37,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:34:37,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814731 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:34:39,804::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:34:39,805::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:34:40,265::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:34:40,272::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:34:40,273::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:34:44,350::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:34:44,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136637 s, 300 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:34:47,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:34:48,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821913 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:34:54,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:34:54,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789912 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:34:58,048::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:34:58,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855387 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:35:04,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:35:04,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110971 s, 369 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:35:08,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:35:08,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000750583 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:35:14,482::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:35:14,511::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861872 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:35:18,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:35:18,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00069052 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:35:24,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:35:24,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141985 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:35:28,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:35:28,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141867 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:35:34,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:35:34,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155846 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:35:38,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:35:38,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159683 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14636::WARNING::2013-03-17 21:35:40,305::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:35:40,312::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 21:35:44,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:35:44,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832611 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:35:48,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:35:48,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837865 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:35:50,314::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 21:35:50,316::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:35:50,317::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:35:50,712::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:35:50,719::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:35:50,720::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:35:54,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:35:54,671::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890239 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:35:58,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:35:58,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014932 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:36:04,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:36:04,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861064 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:36:08,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:36:08,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00506325 s, 809 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:36:14,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:36:14,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148517 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:36:18,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:36:18,698::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000726437 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:36:24,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:36:24,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169427 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:36:28,738::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:36:28,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151416 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:36:34,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:36:34,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179875 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:36:38,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:36:38,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792071 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:36:44,871::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:36:44,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835955 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:36:48,899::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 21:36:48,903::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 21:36:48,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:36:48,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000685804 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:36:48,942::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000685804 s, 3.0 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 21:36:48,942::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 21:36:48,943::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 21:36:48,944::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 21:36:48,945::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:36:48,946::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:36:48,947::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:36:48,947::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:36:48,947::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:36:48,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:36:49,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:36:49,550::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:36:49,552::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 21:36:49,555::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 21:36:49,557::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 21:36:49,557::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 21:36:49,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:36:50,365::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:36:50,372::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:36:50,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0203927 s, 201 kB/s\n'; <rc> = 0
>Thread-14637::WARNING::2013-03-17 21:36:50,748::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:36:50,752::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 21:36:54,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:36:54,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000889322 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:37:00,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:37:00,785::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:37:00,788::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:37:00,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017063 s, 240 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:37:01,230::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:37:01,235::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:37:01,237::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:37:04,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:37:04,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000903548 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:37:10,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:37:10,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816765 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:37:14,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:37:15,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145878 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:37:20,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:37:20,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828076 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:37:25,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:37:25,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170535 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:37:31,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:37:31,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840738 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:37:35,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:37:35,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783799 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:37:41,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:37:41,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00308523 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:37:45,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:37:45,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815763 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:37:51,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:37:51,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136221 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:37:55,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:37:55,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774784 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:38:01,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-14638::WARNING::2013-03-17 21:38:01,288::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:38:01,295::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 21:38:01,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00793251 s, 516 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:38:05,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:38:05,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079812 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:38:11,301::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:38:11,302::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:38:11,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:38:11,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00973576 s, 421 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:38:11,750::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:38:11,756::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:38:11,757::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:38:15,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:38:15,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780014 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:38:21,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:38:21,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0219382 s, 187 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:38:25,281::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:38:25,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826539 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:38:31,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:38:31,607::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108075 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:38:35,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:38:35,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112947 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:38:41,647::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:38:41,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891079 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:38:45,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:38:45,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786454 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:38:51,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:38:51,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839181 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:38:55,390::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 21:38:55,391::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 21:38:55,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:38:55,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00129684 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:38:55,434::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00129684 s, 1.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 21:38:55,435::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 21:38:55,435::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 21:38:55,437::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 21:38:55,438::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:38:55,438::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:38:55,439::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:38:55,439::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:38:55,439::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:38:55,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:38:55,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:38:55,864::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:38:55,865::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 21:38:55,868::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 21:38:55,869::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 21:38:55,870::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 21:38:55,871::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:38:56,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:38:56,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:38:56,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188193 s, 218 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:39:01,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:39:01,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837547 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:39:06,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:39:06,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133745 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14639::WARNING::2013-03-17 21:39:11,785::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:39:11,789::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 21:39:11,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:39:11,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870947 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:39:16,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:39:16,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857812 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:39:21,791::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:39:21,792::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:39:21,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:39:22,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147581 s, 2.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:39:22,293::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:39:22,299::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:39:22,301::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:39:26,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:39:26,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000717059 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:39:32,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:39:32,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747748 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:39:36,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:39:36,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856034 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:39:42,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:39:42,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00326261 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:39:46,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:39:46,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139778 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:39:52,227::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:39:52,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832816 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:39:56,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:39:56,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825748 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:40:02,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:40:02,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164368 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:40:06,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:40:06,569::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827038 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:40:12,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:40:12,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143343 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:40:16,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:40:16,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144763 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14640::WARNING::2013-03-17 21:40:22,336::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:40:22,339::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 21:40:22,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:40:22,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859328 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:40:26,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:40:26,649::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839035 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:40:32,348::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:40:32,349::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:40:32,528::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:40:32,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168983 s, 242 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:40:32,830::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:40:32,835::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:40:32,837::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:40:36,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:40:36,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883162 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:40:42,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:40:42,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837385 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:40:46,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:40:46,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161167 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:40:52,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:40:52,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161262 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:40:56,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:40:56,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081897 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:41:02,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:41:02,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00265448 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:41:06,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:41:06,793::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139177 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:41:12,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:41:12,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844244 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:41:16,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:41:16,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167426 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:41:22,941::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:41:22,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136735 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:41:26,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:41:26,881::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171159 s, 2.4 MB/s\n'; <rc> = 0
>Thread-14641::WARNING::2013-03-17 21:41:32,866::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:41:32,869::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 21:41:33,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:41:33,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812215 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:41:36,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:41:36,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781515 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:41:42,871::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 21:41:42,872::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:41:42,873::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:41:43,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:41:43,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144183 s, 284 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:41:43,411::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:41:43,417::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:41:43,419::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:41:46,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:41:46,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746858 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:41:53,197::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 21:41:53,199::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 21:41:53,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:41:53,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0011025 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:41:53,232::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0011025 s, 1.9 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 21:41:53,233::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 21:41:53,233::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 21:41:53,234::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 21:41:53,235::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:41:53,236::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:41:53,237::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:41:53,237::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:41:53,237::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:41:53,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:41:53,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:41:53,699::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:41:53,700::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 21:41:53,703::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 21:41:53,706::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 21:41:53,706::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 21:41:53,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:41:54,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:41:54,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:41:54,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159957 s, 256 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:41:56,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:41:56,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000905704 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:42:04,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:42:04,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126012 s, 325 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:42:06,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:42:07,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160067 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:42:14,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:42:14,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835096 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:42:17,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:42:17,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000927854 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:42:24,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:42:24,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00095008 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:42:27,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:42:27,130::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140011 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:42:34,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:42:34,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153008 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:42:37,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:42:37,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881085 s, 4.6 MB/s\n'; <rc> = 0
>Thread-14642::WARNING::2013-03-17 21:42:43,445::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:42:43,449::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 21:42:44,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:42:44,526::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815318 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:42:47,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:42:47,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788332 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:42:53,451::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:42:53,453::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:42:53,937::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:42:53,944::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:42:53,945::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:42:54,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:42:54,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917717 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:42:57,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:42:57,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795961 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:43:04,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:43:04,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000696458 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:43:07,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:43:07,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135362 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:43:14,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:43:14,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084464 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:43:17,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:43:17,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799716 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:43:24,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:43:24,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826769 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:43:27,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:43:27,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139135 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:43:34,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:43:34,928::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139926 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:43:37,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:43:37,404::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148237 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:43:44,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:43:45,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149237 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:43:47,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:43:47,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151705 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14643::WARNING::2013-03-17 21:43:53,976::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:43:53,979::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 21:43:55,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:43:55,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837448 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:43:57,453::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 21:43:57,456::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 21:43:57,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:43:57,483::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000804325 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:43:57,485::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000804325 s, 2.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 21:43:57,485::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 21:43:57,485::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 21:43:57,487::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 21:43:57,488::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:43:57,489::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:43:57,489::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:43:57,490::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:43:57,490::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:43:57,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:43:57,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:43:57,908::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:43:57,909::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 21:43:57,911::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 21:43:57,912::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 21:43:57,913::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 21:43:57,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:43:58,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:43:58,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:43:58,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00470661 s, 870 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:44:03,981::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:44:03,983::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:44:04,383::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:44:04,393::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:44:04,394::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:44:05,118::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:44:05,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091602 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:44:08,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:44:08,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909446 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:44:15,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:44:15,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884929 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:44:18,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:44:18,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846618 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:44:25,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:44:25,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886174 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:44:28,395::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:44:28,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826612 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:44:35,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:44:35,350::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858286 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:44:38,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:44:38,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157844 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:44:45,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:44:45,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824465 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:44:48,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:44:48,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143293 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:44:55,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:44:55,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074528 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:44:58,532::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:44:58,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170762 s, 2.4 MB/s\n'; <rc> = 0
>Thread-14644::WARNING::2013-03-17 21:45:04,438::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:45:04,445::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 21:45:05,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:45:05,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728556 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:45:08,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:45:08,607::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000994414 s, 4.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:45:14,447::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:45:14,449::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:45:14,911::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:45:14,918::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:45:14,919::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:45:15,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:45:15,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169744 s, 241 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:45:18,614::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:45:18,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814222 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:45:25,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:45:25,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893446 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:45:28,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:45:28,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911021 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:45:35,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:45:35,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778109 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:45:38,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:45:38,712::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142498 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:45:45,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:45:45,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165306 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:45:48,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:45:48,756::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137116 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:45:55,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:45:55,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168765 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:45:58,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:45:58,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140622 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:46:05,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:46:06,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143498 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:46:08,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:46:08,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0024399 s, 1.7 MB/s\n'; <rc> = 0
>Thread-14645::WARNING::2013-03-17 21:46:14,946::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:46:14,950::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 21:46:16,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:46:16,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912412 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:46:18,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:46:18,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898426 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:46:24,952::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:46:24,954::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:46:25,387::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:46:25,393::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:46:25,395::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:46:26,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:46:26,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171745 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:46:28,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:46:28,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751535 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:46:36,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:46:36,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000715456 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:46:38,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:46:38,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144497 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:46:46,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:46:46,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847606 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:46:48,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:46:48,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818756 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:46:56,356::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 21:46:56,358::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 21:46:56,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:46:56,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00133212 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:46:56,401::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00133212 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 21:46:56,402::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 21:46:56,402::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 21:46:56,403::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 21:46:56,404::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:46:56,405::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:46:56,406::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:46:56,406::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:46:56,406::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:46:56,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:46:56,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:46:56,806::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:46:56,808::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 21:46:56,812::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 21:46:56,813::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 21:46:56,814::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 21:46:56,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:46:57,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:46:57,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:46:57,309::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112596 s, 364 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:46:59,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:46:59,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145543 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:47:07,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:47:07,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168711 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:47:09,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:47:09,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165257 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:47:17,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:47:17,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000969586 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:47:19,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:47:19,140::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162798 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14646::WARNING::2013-03-17 21:47:25,420::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:47:25,424::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 21:47:27,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:47:27,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812278 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:47:29,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:47:29,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844112 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:47:35,426::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 21:47:35,427::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:47:35,428::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:47:35,870::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:47:35,876::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:47:35,877::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:47:37,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:47:37,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101285 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:47:39,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:47:39,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143549 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:47:47,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:47:47,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830875 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:47:49,230::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:47:49,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133899 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:47:57,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:47:57,801::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862349 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:47:59,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:47:59,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00192044 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:48:07,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:48:07,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138903 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:48:09,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:48:09,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000926172 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:48:17,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:48:17,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793323 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:48:19,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:48:19,402::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166015 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:48:27,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:48:28,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00094657 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:48:29,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:48:29,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075314 s, 5.4 MB/s\n'; <rc> = 0
>Thread-14647::WARNING::2013-03-17 21:48:35,906::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:48:35,909::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 21:48:38,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:48:38,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895345 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:48:39,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:48:39,481::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852047 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:48:45,911::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:48:45,912::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:48:46,331::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:48:46,338::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:48:46,339::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:48:48,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:48:48,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810164 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:48:49,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:48:49,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000730187 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:48:58,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:48:58,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092239 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:48:59,520::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 21:48:59,522::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 21:48:59,522::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:48:59,552::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000984777 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:48:59,556::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000984777 s, 2.1 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 21:48:59,557::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 21:48:59,557::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 21:48:59,559::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 21:48:59,560::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:48:59,560::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:48:59,561::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:48:59,561::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:48:59,561::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:48:59,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:48:59,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:48:59,998::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:48:59,999::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 21:49:00,001::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 21:49:00,002::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 21:49:00,003::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 21:49:00,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:49:00,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:49:00,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:49:00,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186433 s, 220 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:49:08,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:49:08,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153042 s, 268 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:49:10,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:49:10,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148308 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:49:18,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:49:18,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892006 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:49:20,465::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:49:20,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150615 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:49:28,429::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:49:28,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896464 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:49:30,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:49:30,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775272 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:49:38,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:49:38,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155801 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:49:40,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:49:40,591::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078927 s, 5.2 MB/s\n'; <rc> = 0
>Thread-14648::WARNING::2013-03-17 21:49:46,365::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:49:46,373::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 21:49:48,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:49:48,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886822 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:49:50,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:49:50,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076054 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:49:56,376::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:49:56,377::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:49:56,922::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:49:56,929::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:49:56,930::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:49:58,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:49:58,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080472 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:50:00,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:50:00,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000908911 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:50:08,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:50:08,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891626 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:50:10,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:50:10,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147577 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:50:18,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:50:18,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828572 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:50:20,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:50:20,777::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00094585 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:50:28,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:50:28,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145431 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:50:30,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:50:30,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142393 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:50:38,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:50:38,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151013 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:50:40,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:50:40,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164738 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:50:49,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:50:49,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144244 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:50:50,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:50:50,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083195 s, 4.9 MB/s\n'; <rc> = 0
>Thread-14649::WARNING::2013-03-17 21:50:56,955::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:50:56,958::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 21:50:59,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:50:59,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853948 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:51:00,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:51:00,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106878 s, 3.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:51:06,961::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:51:06,962::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:51:07,371::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:51:07,378::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:51:07,379::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:51:09,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:51:09,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873292 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:51:10,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:51:10,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147678 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:51:19,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:51:19,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146392 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:51:20,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:51:21,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000905103 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:51:29,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:51:29,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001643 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:51:31,021::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:51:31,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165344 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:51:39,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:51:39,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868867 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:51:41,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:51:41,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133313 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:51:49,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:51:49,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154139 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:51:51,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:51:51,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135264 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:51:59,567::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 21:51:59,569::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 21:51:59,571::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:51:59,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00165408 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:51:59,602::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00165408 s, 1.2 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 21:51:59,603::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 21:51:59,603::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 21:51:59,604::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 21:51:59,608::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:51:59,609::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:51:59,609::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:51:59,610::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:51:59,610::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:51:59,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:52:00,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:52:00,141::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:52:00,142::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 21:52:00,144::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 21:52:00,145::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 21:52:00,146::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 21:52:00,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:52:00,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:52:00,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:52:00,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0215093 s, 190 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:52:01,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:52:01,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132613 s, 309 kB/s\n'; <rc> = 0
>Thread-14650::WARNING::2013-03-17 21:52:07,405::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:52:07,409::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 21:52:10,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:52:10,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747595 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:52:11,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:52:11,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790792 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:52:17,411::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:52:17,413::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:52:17,814::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:52:17,821::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:52:17,823::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:52:20,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:52:20,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106738 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:52:21,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:52:21,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839774 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:52:30,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:52:30,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142715 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:52:31,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:52:31,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154159 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:52:40,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:52:40,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821744 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:52:41,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:52:41,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144359 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:52:50,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:52:50,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123766 s, 331 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:52:51,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:52:51,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145152 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:53:01,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:53:01,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000992175 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:53:01,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:53:01,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145198 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:53:11,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:53:11,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175547 s, 233 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:53:11,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:53:11,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777514 s, 5.3 MB/s\n'; <rc> = 0
>Thread-14651::WARNING::2013-03-17 21:53:17,856::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:53:17,860::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 21:53:21,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:53:21,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891733 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:53:21,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:53:21,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00196538 s, 2.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:53:27,861::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 21:53:27,862::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:53:27,863::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:53:28,249::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:53:28,256::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:53:28,257::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:53:31,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:53:31,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907834 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:53:31,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:53:31,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146441 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:53:41,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:53:41,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909289 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:53:41,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:53:41,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00189331 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:53:51,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:53:51,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816741 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:53:51,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:53:51,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135887 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:54:01,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:54:01,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075659 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:54:01,670::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 21:54:01,671::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 21:54:01,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:54:01,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00532055 s, 385 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:54:01,733::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00532055 s, 385 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 21:54:01,734::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 21:54:01,734::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 21:54:01,736::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 21:54:01,737::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:54:01,738::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:54:01,738::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:54:01,739::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:54:01,739::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:54:01,740::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:54:02,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:54:02,271::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:54:02,272::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 21:54:02,275::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 21:54:02,276::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 21:54:02,276::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 21:54:02,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:54:02,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:54:02,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:54:02,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015012 s, 273 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:54:11,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:54:11,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139019 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:54:12,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:54:12,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898064 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:54:21,625::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:54:21,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000716031 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:54:22,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:54:22,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139391 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14652::WARNING::2013-03-17 21:54:28,290::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:54:28,295::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 21:54:31,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:54:31,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185025 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:54:32,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:54:32,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828122 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:54:38,301::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:54:38,302::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:54:38,701::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:54:38,708::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:54:38,710::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:54:41,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:54:41,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873635 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:54:42,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:54:42,819::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766474 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:54:51,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:54:51,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150884 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:54:52,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:54:52,866::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161642 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:55:01,941::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:55:01,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842287 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:55:02,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:55:02,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802995 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:55:12,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:55:12,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00180963 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:55:12,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:55:12,944::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133782 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:55:22,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:55:22,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148873 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:55:22,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:55:22,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160283 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:55:32,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:55:32,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923323 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:55:32,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:55:33,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136752 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14653::WARNING::2013-03-17 21:55:38,736::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:55:38,739::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 21:55:42,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:55:42,297::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923639 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:55:43,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:55:43,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846994 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:55:48,741::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:55:48,743::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:55:49,142::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:55:49,149::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:55:49,151::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:55:52,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:55:52,365::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841318 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:55:53,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:55:53,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862295 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:56:02,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:56:02,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815936 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:56:03,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:56:03,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0241652 s, 170 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:56:12,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:56:12,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862276 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:56:13,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:56:13,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080674 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:56:22,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:56:22,602::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146962 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:56:23,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:56:23,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080547 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:56:32,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:56:32,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782281 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:56:33,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:56:33,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00248682 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:56:42,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:56:42,744::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000983052 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:56:43,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:56:43,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013934 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14654::WARNING::2013-03-17 21:56:49,185::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:56:49,190::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 21:56:52,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:56:52,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872664 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:56:53,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:56:53,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128217 s, 3.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:56:59,195::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:56:59,196::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:56:59,633::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:56:59,639::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:56:59,640::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:57:02,856::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 21:57:02,859::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 21:57:02,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:57:02,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00140403 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:57:02,897::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00140403 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 21:57:02,898::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 21:57:02,898::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 21:57:02,900::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 21:57:02,901::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:57:02,901::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:57:02,902::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:57:02,902::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:57:02,902::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 21:57:02,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:57:03,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:57:03,327::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:57:03,328::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 21:57:03,330::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 21:57:03,331::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 21:57:03,332::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 21:57:03,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:57:03,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:57:03,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00744637 s, 550 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:57:03,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:57:03,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:57:03,818::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00765643 s, 535 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:57:13,490::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:57:13,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115878 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:57:13,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:57:13,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780742 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:57:23,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:57:23,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863426 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:57:23,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:57:23,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801453 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:57:33,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:57:33,649::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177542 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:57:34,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:57:34,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144981 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:57:43,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:57:43,712::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000982513 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:57:44,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:57:44,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118354 s, 346 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:57:53,717::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:57:53,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115804 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:57:54,178::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:57:54,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114423 s, 358 kB/s\n'; <rc> = 0
>Thread-14655::WARNING::2013-03-17 21:57:59,671::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:57:59,677::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 21:58:03,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:58:03,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125666 s, 326 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:58:04,271::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:58:04,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868141 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:58:09,680::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:58:09,681::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:58:10,149::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:58:10,156::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:58:10,157::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:58:13,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:58:13,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776539 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:58:14,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:58:14,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134097 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:58:23,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:58:23,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174437 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:58:24,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:58:24,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.009767 s, 419 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:58:33,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:58:33,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141781 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:58:34,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:58:34,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167677 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:58:43,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:58:44,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107 s, 383 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:58:44,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:58:44,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153422 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:58:54,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:58:54,081::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016493 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:58:54,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:58:54,698::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844434 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:59:04,085::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 21:59:04,088::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 21:59:04,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:59:04,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00923854 s, 222 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:59:04,128::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00923854 s, 222 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 21:59:04,129::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 21:59:04,129::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 21:59:04,132::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 21:59:04,133::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:59:04,133::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:59:04,134::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:59:04,134::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:59:04,135::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 21:59:04,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:59:04,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:59:04,589::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 21:59:04,590::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 21:59:04,592::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 21:59:04,593::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 21:59:04,594::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 21:59:04,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:59:04,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:59:04,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136378 s, 300 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:59:05,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:59:05,045::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:59:05,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185998 s, 220 kB/s\n'; <rc> = 0
>Thread-14656::WARNING::2013-03-17 21:59:10,188::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 21:59:10,193::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 21:59:14,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:59:14,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913014 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:59:15,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:59:15,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000962394 s, 4.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 21:59:20,195::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 21:59:20,197::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 21:59:20,198::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 21:59:20,629::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 21:59:20,636::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 21:59:20,638::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 21:59:24,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:59:24,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816533 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:59:25,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:59:25,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765397 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:59:34,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:59:35,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015269 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:59:35,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:59:35,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887134 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:59:45,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:59:45,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139483 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:59:45,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:59:45,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079002 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 21:59:55,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 21:59:55,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000736714 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 21:59:55,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 21:59:55,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859778 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:00:05,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:00:05,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:00:05,347::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010297 s, 398 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:00:05,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101653 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:00:15,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:00:15,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:00:15,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112567 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:00:15,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141968 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14657::WARNING::2013-03-17 22:00:20,668::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:00:20,671::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:00:25,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:00:25,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791871 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:00:25,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:00:25,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815811 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:00:30,677::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:00:30,678::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:00:31,098::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:00:31,105::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:00:31,106::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:00:35,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:00:35,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014893 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:00:35,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:00:35,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791313 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:00:45,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:00:45,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077517 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:00:45,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:00:45,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017117 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:00:55,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:00:55,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00227701 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:00:55,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:00:55,791::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812497 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:01:05,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:01:05,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158028 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:01:05,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:01:05,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157637 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:01:15,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:01:15,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177146 s, 231 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:01:15,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:01:15,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115805 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:01:25,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:01:25,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137621 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:01:25,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:01:26,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875117 s, 4.7 MB/s\n'; <rc> = 0
>Thread-14658::WARNING::2013-03-17 22:01:31,132::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:01:31,136::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:01:35,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:01:35,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00095384 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:01:36,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:01:36,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00201378 s, 2.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:01:41,140::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:01:41,141::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:01:41,580::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:01:41,587::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:01:41,588::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:01:45,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:01:45,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151797 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:01:46,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:01:46,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823519 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:01:55,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:01:55,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827973 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:01:56,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:01:56,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165201 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:02:05,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:02:05,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00271824 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:02:06,305::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 22:02:06,307::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 22:02:06,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:02:06,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0008569 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:02:06,341::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0008569 s, 2.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 22:02:06,341::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 22:02:06,342::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 22:02:06,344::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 22:02:06,345::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:02:06,345::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:02:06,346::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:02:06,346::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:02:06,347::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:02:06,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:02:06,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:02:06,802::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:02:06,803::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 22:02:06,806::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 22:02:06,807::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 22:02:06,807::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 22:02:06,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:02:07,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:02:07,156::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:02:07,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132503 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:02:15,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:02:16,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923433 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:02:17,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:02:17,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162762 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:02:26,014::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:02:26,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157949 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:02:27,312::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:02:27,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000972102 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:02:36,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:02:36,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00330456 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:02:37,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:02:37,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086558 s, 4.7 MB/s\n'; <rc> = 0
>Thread-14659::WARNING::2013-03-17 22:02:41,615::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:02:41,618::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:02:46,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:02:46,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082339 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:02:47,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:02:47,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164246 s, 2.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:02:51,620::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:02:51,621::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:02:52,056::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:02:52,063::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:02:52,064::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:02:56,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:02:56,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838952 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:02:57,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:02:57,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870287 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:03:06,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:03:06,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164504 s, 249 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:03:07,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:03:07,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171789 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:03:16,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:03:16,287::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151316 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:03:17,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:03:17,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127699 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:03:26,292::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:03:26,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911906 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:03:27,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:03:27,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000998689 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:03:36,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:03:36,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154354 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:03:37,837::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:03:37,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170266 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:03:46,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:03:46,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101838 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:03:47,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:03:47,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765593 s, 5.4 MB/s\n'; <rc> = 0
>Thread-14660::WARNING::2013-03-17 22:03:52,089::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:03:52,092::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:03:56,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:03:56,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892024 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:03:57,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:03:58,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000989489 s, 4.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:04:02,094::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:04:02,096::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:04:02,556::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:04:02,562::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:04:02,564::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:04:06,454::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 22:04:06,457::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 22:04:06,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:04:06,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00153085 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:04:06,497::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00153085 s, 1.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 22:04:06,498::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 22:04:06,498::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 22:04:06,499::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 22:04:06,501::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:04:06,501::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:04:06,502::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:04:06,502::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:04:06,502::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:04:06,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:04:06,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:04:06,914::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:04:06,915::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 22:04:06,918::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 22:04:06,919::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 22:04:06,919::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 22:04:06,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:04:07,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:04:07,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:04:07,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0317966 s, 129 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:04:08,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:04:08,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0069208 s, 592 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:04:17,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:04:17,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136889 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:04:18,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:04:18,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832502 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:04:27,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:04:27,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862565 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:04:28,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:04:28,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000707758 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:04:37,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:04:37,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112329 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:04:38,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:04:38,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153865 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:04:47,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:04:47,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803359 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:04:48,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:04:48,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083374 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:04:57,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:04:57,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166873 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:04:58,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:04:58,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840369 s, 4.9 MB/s\n'; <rc> = 0
>Thread-14661::WARNING::2013-03-17 22:05:02,591::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:05:02,597::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:05:07,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:05:07,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806055 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:05:08,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:05:08,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847484 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:05:12,600::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 22:05:12,601::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:05:12,601::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:05:13,037::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:05:13,044::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:05:13,046::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:05:17,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:05:17,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192802 s, 212 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:05:18,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:05:18,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761652 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:05:27,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:05:27,756::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852082 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:05:28,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:05:28,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740987 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:05:37,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:05:37,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147622 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:05:38,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:05:38,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853311 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:05:47,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:05:47,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777364 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:05:48,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:05:48,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105916 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:05:57,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:05:57,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000963385 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:05:58,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:05:58,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00417839 s, 980 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:06:07,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:06:07,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754915 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:06:08,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:06:08,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013375 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14662::WARNING::2013-03-17 22:06:13,070::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:06:13,074::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:06:18,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:06:18,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831022 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:06:19,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:06:19,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000733752 s, 5.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:06:23,077::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:06:23,078::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:06:23,573::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:06:23,579::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:06:23,581::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:06:28,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:06:28,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00283033 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:06:29,108::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:06:29,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000960864 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:06:38,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:06:38,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000729917 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:06:39,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:06:39,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847413 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:06:48,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:06:48,311::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862158 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:06:49,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:06:49,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152257 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:06:58,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:06:58,401::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160994 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:06:59,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:06:59,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174543 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:07:08,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:07:08,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000713886 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:07:09,412::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 22:07:09,414::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 22:07:09,415::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:07:09,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00147914 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:07:09,458::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00147914 s, 1.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 22:07:09,459::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 22:07:09,459::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 22:07:09,461::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 22:07:09,463::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:07:09,464::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:07:09,464::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:07:09,464::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:07:09,465::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:07:09,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:07:09,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:07:09,900::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:07:09,901::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 22:07:09,903::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 22:07:09,905::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 22:07:09,905::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 22:07:09,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:07:10,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:07:10,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:07:10,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016594 s, 247 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:07:18,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:07:18,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845765 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:07:20,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:07:20,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767711 s, 5.3 MB/s\n'; <rc> = 0
>Thread-14663::WARNING::2013-03-17 22:07:23,605::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:07:23,609::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:07:28,524::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:07:28,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117335 s, 349 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:07:30,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:07:30,513::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832507 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:07:33,612::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:07:33,613::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:07:34,061::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:07:34,068::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:07:34,070::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:07:38,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:07:38,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152087 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:07:40,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:07:40,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154171 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:07:48,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:07:48,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134486 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:07:50,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:07:50,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175515 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:07:58,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:07:58,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141599 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:08:00,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:08:00,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084587 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:08:08,730::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:08:08,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761464 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:08:10,797::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:08:10,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820646 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:08:18,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:08:18,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916927 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:08:20,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:08:20,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145087 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:08:28,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:08:28,843::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154822 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:08:30,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:08:30,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144622 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14664::WARNING::2013-03-17 22:08:34,104::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:08:34,108::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:08:38,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:08:38,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773098 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:08:41,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:08:41,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872241 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:08:44,110::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:08:44,111::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:08:44,521::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:08:44,528::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:08:44,529::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:08:48,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:08:48,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916112 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:08:51,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:08:51,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854181 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:08:58,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:08:58,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154176 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:09:01,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:09:01,195::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819937 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:09:08,962::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 22:09:08,963::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 22:09:08,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:09:09,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00155602 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:09:09,003::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00155602 s, 1.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 22:09:09,004::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 22:09:09,004::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 22:09:09,006::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 22:09:09,007::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:09:09,007::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:09:09,008::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:09:09,008::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:09:09,009::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:09:09,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:09:09,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:09:09,440::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:09:09,441::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 22:09:09,444::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 22:09:09,445::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 22:09:09,445::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 22:09:09,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:09:09,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:09:09,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:09:09,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155111 s, 264 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:09:11,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:09:11,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017679 s, 232 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:09:19,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:09:19,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875435 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:09:21,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:09:21,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147618 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:09:29,944::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:09:29,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152136 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:09:31,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:09:31,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155759 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:09:39,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:09:40,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844068 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:09:41,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:09:41,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173188 s, 2.4 MB/s\n'; <rc> = 0
>Thread-14665::WARNING::2013-03-17 22:09:44,559::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:09:44,562::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:09:50,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:09:50,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157582 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:09:51,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:09:51,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121907 s, 3.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:09:54,563::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:09:54,564::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:09:55,019::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:09:55,026::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:09:55,027::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:10:00,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:10:00,195::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143386 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:10:01,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:10:01,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00401416 s, 1.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:10:10,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:10:10,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00318727 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:10:11,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:10:11,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000940289 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:10:20,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:10:20,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863652 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:10:21,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:10:21,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154157 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:10:30,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:10:30,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811971 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:10:31,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:10:31,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851316 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:10:40,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:10:40,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156609 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:10:42,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:10:42,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000956744 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:10:50,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:10:50,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157126 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:10:52,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:10:52,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160495 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14666::WARNING::2013-03-17 22:10:55,054::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:10:55,058::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:11:00,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:11:00,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835333 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:11:02,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:11:02,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000927042 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:11:05,069::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 22:11:05,070::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:11:05,071::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:11:05,503::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:11:05,510::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:11:05,511::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:11:10,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:11:10,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145082 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:11:12,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:11:12,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148927 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:11:20,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:11:20,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000705265 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:11:22,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:11:22,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147146 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:11:30,532::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:11:30,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770329 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:11:32,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:11:32,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000745543 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:11:40,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:11:40,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856516 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:11:42,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:11:42,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826016 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:11:50,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:11:50,631::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774622 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:11:52,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:11:52,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749116 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:12:00,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:12:00,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144086 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:12:02,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:12:02,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163067 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14667::WARNING::2013-03-17 22:12:05,535::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:12:05,539::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:12:10,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:12:10,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796957 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:12:12,748::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 22:12:12,749::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 22:12:12,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:12:12,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000767506 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:12:12,784::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000767506 s, 2.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 22:12:12,784::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 22:12:12,784::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 22:12:12,786::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 22:12:12,788::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:12:12,788::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:12:12,789::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:12:12,789::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:12:12,790::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:12:12,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:12:13,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:12:13,206::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:12:13,207::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 22:12:13,210::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 22:12:13,211::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 22:12:13,211::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 22:12:13,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:12:13,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:12:13,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:12:13,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00730073 s, 561 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:12:15,541::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:12:15,542::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:12:15,964::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:12:15,971::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:12:15,972::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:12:20,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:12:20,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101969 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:12:23,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:12:23,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883127 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:12:30,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:12:30,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796008 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:12:33,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:12:33,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886171 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:12:40,796::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:12:40,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015188 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:12:43,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:12:43,820::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857384 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:12:50,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:12:50,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786354 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:12:53,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:12:53,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740469 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:13:00,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:13:00,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157595 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:13:03,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:13:03,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917309 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:13:10,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:13:10,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764974 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:13:14,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:13:14,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162373 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14668::WARNING::2013-03-17 22:13:16,001::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:13:16,005::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:13:20,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:13:20,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824699 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:13:24,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:13:24,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100141 s, 409 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:13:26,007::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:13:26,009::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:13:26,447::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:13:26,454::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:13:26,455::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:13:31,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:13:31,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121202 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:13:34,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:13:34,195::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854006 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:13:41,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:13:41,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135349 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:13:44,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:13:44,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0029965 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:13:51,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:13:51,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826633 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:13:54,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:13:54,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00069261 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:14:01,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:14:01,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132561 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:14:04,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:14:04,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168875 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:14:11,158::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 22:14:11,160::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 22:14:11,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:14:11,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00069519 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:14:11,194::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00069519 s, 2.9 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 22:14:11,194::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 22:14:11,195::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 22:14:11,197::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 22:14:11,197::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:14:11,198::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:14:11,198::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:14:11,199::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:14:11,199::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:14:11,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:14:11,602::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:14:11,607::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:14:11,608::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 22:14:11,613::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 22:14:11,614::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 22:14:11,614::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 22:14:11,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:14:11,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:14:11,997::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:14:12,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182572 s, 224 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:14:14,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:14:14,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000921498 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:14:22,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:14:22,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157658 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:14:24,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:14:24,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740256 s, 5.5 MB/s\n'; <rc> = 0
>Thread-14669::WARNING::2013-03-17 22:14:26,494::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:14:26,497::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:14:32,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:14:32,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898188 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:14:34,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:14:34,667::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879221 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:14:36,499::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:14:36,500::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:14:36,955::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:14:36,962::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:14:36,963::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:14:42,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:14:42,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00269121 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:14:44,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:14:44,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141706 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:14:52,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:14:52,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798776 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:14:54,781::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:14:54,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175886 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:15:02,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:15:02,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778291 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:15:04,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:15:04,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841453 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:15:12,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:15:12,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143628 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:15:14,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:15:14,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789131 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:15:22,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:15:22,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154624 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:15:25,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:15:25,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0344521 s, 119 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:15:32,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:15:32,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129296 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:15:35,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:15:35,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111792 s, 3.7 MB/s\n'; <rc> = 0
>Thread-14670::WARNING::2013-03-17 22:15:36,994::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:15:36,997::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:15:42,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:15:42,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000935877 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:15:45,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:15:45,266::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871103 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:15:46,999::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:15:47,000::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:15:47,448::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:15:47,455::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:15:47,457::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:15:52,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:15:52,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000955067 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:15:55,305::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:15:55,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849065 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:16:02,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:16:02,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154211 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:16:05,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:16:05,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147342 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:16:12,528::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:16:12,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000964352 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:16:15,471::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:16:15,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137323 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:16:22,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:16:22,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858452 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:16:25,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:16:25,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143477 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:16:32,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:16:32,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140555 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:16:35,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:16:35,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144517 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:16:42,648::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:16:42,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808458 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:16:45,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:16:45,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015919 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14671::WARNING::2013-03-17 22:16:47,492::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:16:47,496::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:16:52,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:16:52,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804547 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:16:55,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:16:55,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836492 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:16:57,497::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 22:16:57,500::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:16:57,501::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:16:57,927::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:16:57,934::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:16:57,935::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:17:02,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:17:02,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845497 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:17:05,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:17:05,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013845 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:17:12,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:17:12,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081592 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:17:15,948::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 22:17:15,949::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 22:17:15,950::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:17:15,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000783255 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:17:15,985::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000783255 s, 2.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 22:17:15,986::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 22:17:15,986::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 22:17:15,988::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 22:17:15,990::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:17:15,991::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:17:15,991::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:17:15,992::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:17:15,992::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:17:15,993::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:17:16,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:17:16,380::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:17:16,381::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 22:17:16,384::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 22:17:16,385::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 22:17:16,385::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 22:17:16,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:17:16,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:17:16,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:17:16,777::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159493 s, 257 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:17:22,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:17:22,837::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000952184 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:17:26,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:17:26,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753577 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:17:32,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:17:32,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804412 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:17:36,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:17:36,936::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143311 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:17:42,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:17:42,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000700548 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:17:46,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:17:47,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000697567 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:17:52,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:17:52,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157231 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:17:57,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:17:57,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.033501 s, 122 kB/s\n'; <rc> = 0
>Thread-14672::WARNING::2013-03-17 22:17:57,965::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:17:57,970::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:18:02,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:18:02,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839964 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:18:07,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:18:07,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761877 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:18:08,002::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:18:08,003::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:18:08,416::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:18:08,423::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:18:08,424::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:18:13,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:18:13,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000715823 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:18:17,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:18:17,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792599 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:18:23,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:18:23,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159145 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:18:27,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:18:27,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154648 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:18:33,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:18:33,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149322 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:18:37,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:18:37,756::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010035 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:18:43,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:18:43,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146672 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:18:47,796::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:18:47,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785329 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:18:53,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:18:53,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133322 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:18:57,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:18:57,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156437 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:19:03,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:19:03,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0526939 s, 77.7 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:19:07,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:19:08,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000940237 s, 4.4 MB/s\n'; <rc> = 0
>Thread-14673::WARNING::2013-03-17 22:19:08,452::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:19:08,458::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:19:13,314::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 22:19:13,315::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 22:19:13,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:19:13,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000730821 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:19:13,343::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000730821 s, 2.8 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 22:19:13,343::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 22:19:13,343::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 22:19:13,345::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 22:19:13,346::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:19:13,347::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:19:13,347::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:19:13,348::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:19:13,348::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:19:13,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:19:13,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:19:13,794::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:19:13,795::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 22:19:13,798::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 22:19:13,799::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 22:19:13,799::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 22:19:13,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:19:14,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:19:14,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:19:14,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0209992 s, 195 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:19:18,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:19:18,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0215704 s, 190 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:19:18,466::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:19:18,467::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:19:18,888::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:19:18,896::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:19:18,898::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:19:24,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:19:24,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00979003 s, 418 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:19:28,439::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:19:28,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810968 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:19:34,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:19:34,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115942 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:19:38,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:19:38,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785655 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:19:44,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:19:44,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109334 s, 375 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:19:48,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:19:48,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137263 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:19:54,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:19:54,395::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00204222 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:19:58,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:19:58,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142559 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:20:04,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:20:04,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016628 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:20:08,730::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:20:08,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000942047 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:20:14,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:20:14,490::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141049 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:20:18,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:20:18,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000750403 s, 5.5 MB/s\n'; <rc> = 0
>Thread-14674::WARNING::2013-03-17 22:20:18,930::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:20:18,941::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:20:24,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:20:24,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757877 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:20:28,956::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:20:28,957::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:20:28,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:20:29,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00933382 s, 439 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:20:29,417::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:20:29,422::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:20:29,423::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:20:34,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:20:34,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820701 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:20:39,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:20:39,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900933 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:20:44,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:20:44,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100968 s, 406 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:20:49,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:20:49,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883497 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:20:54,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:20:54,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081721 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:20:59,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:20:59,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108668 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:21:04,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:21:04,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146075 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:21:09,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:21:09,347::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154026 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:21:14,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:21:14,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000687547 s, 6.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:21:19,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:21:19,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166949 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:21:24,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:21:24,820::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.025488 s, 161 kB/s\n'; <rc> = 0
>Thread-14675::WARNING::2013-03-17 22:21:29,451::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:21:29,456::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 22:21:29,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:21:29,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823079 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:21:34,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:21:34,881::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0227776 s, 180 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:21:39,458::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:21:39,459::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:21:39,528::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:21:39,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175869 s, 2.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:21:39,925::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:21:39,930::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:21:39,932::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:21:44,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:21:44,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812481 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:21:49,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:21:49,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141329 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:21:54,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:21:54,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844851 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:21:59,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:21:59,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000724404 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:22:04,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:22:04,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819769 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:22:09,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:22:09,805::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086635 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:22:14,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:22:15,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00379912 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:22:19,843::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 22:22:19,845::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 22:22:19,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:22:19,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000985291 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:22:19,892::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000985291 s, 2.1 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 22:22:19,894::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 22:22:19,894::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 22:22:19,896::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 22:22:19,897::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:22:19,897::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:22:19,898::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:22:19,898::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:22:19,899::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:22:19,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:22:20,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:22:20,347::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:22:20,348::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 22:22:20,350::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 22:22:20,351::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 22:22:20,352::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 22:22:20,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:22:20,733::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:22:20,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:22:20,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00865821 s, 473 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:22:25,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:22:25,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141665 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:22:30,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:22:30,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014053 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:22:35,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:22:35,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088236 s, 4.6 MB/s\n'; <rc> = 0
>Thread-14676::WARNING::2013-03-17 22:22:39,958::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:22:39,962::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 22:22:40,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:22:40,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156909 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:22:45,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:22:45,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825685 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:22:49,963::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 22:22:49,965::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:22:49,966::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:22:50,488::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:22:50,495::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:22:50,496::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:22:50,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:22:51,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00873286 s, 469 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:22:55,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:22:55,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854382 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:23:01,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:23:01,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151288 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:23:05,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:23:05,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084794 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:23:11,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:23:11,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170016 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:23:15,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:23:15,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079278 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:23:21,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:23:21,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860293 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:23:25,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:23:25,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149421 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:23:31,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:23:31,311::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754052 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:23:35,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:23:35,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159991 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:23:41,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:23:41,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925216 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:23:45,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:23:45,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163076 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14677::WARNING::2013-03-17 22:23:50,519::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:23:50,523::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 22:23:51,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:23:51,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157698 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:23:55,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:23:55,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840471 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:24:00,525::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:24:00,527::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:24:00,937::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:24:00,945::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:24:00,946::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:24:01,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:24:01,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146903 s, 279 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:24:05,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:24:05,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803617 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:24:11,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:24:11,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757287 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:24:15,450::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 22:24:15,451::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 22:24:15,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:24:15,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000793312 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:24:15,479::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000793312 s, 2.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 22:24:15,479::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 22:24:15,479::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 22:24:15,483::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 22:24:15,484::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:24:15,485::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:24:15,485::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:24:15,486::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:24:15,486::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:24:15,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:24:15,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:24:15,921::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:24:15,922::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 22:24:15,924::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 22:24:15,925::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 22:24:15,926::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 22:24:15,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:24:16,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:24:16,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:24:16,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144858 s, 283 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:24:21,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:24:21,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171729 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:24:26,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:24:26,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169096 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:24:31,863::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:24:31,899::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838735 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:24:36,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:24:36,408::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148747 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:24:41,938::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:24:41,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131005 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:24:46,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:24:46,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876009 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:24:52,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:24:52,046::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867872 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:24:56,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:24:56,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124257 s, 3.3 MB/s\n'; <rc> = 0
>Thread-14678::WARNING::2013-03-17 22:25:00,973::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:25:00,976::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 22:25:02,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:25:02,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803887 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:25:06,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:25:06,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898232 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:25:10,978::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:25:10,980::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:25:11,372::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:25:11,379::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:25:11,380::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:25:12,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:25:12,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141524 s, 289 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:25:16,532::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:25:16,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868285 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:25:22,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:25:22,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808582 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:25:26,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:25:26,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000729932 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:25:32,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:25:32,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156867 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:25:36,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:25:36,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155906 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:25:42,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:25:42,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145114 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:25:46,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:25:46,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763096 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:25:52,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:25:52,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146936 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:25:56,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:25:56,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778866 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:26:02,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:26:02,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140975 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:26:06,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:26:06,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101008 s, 4.1 MB/s\n'; <rc> = 0
>Thread-14679::WARNING::2013-03-17 22:26:11,407::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:26:11,411::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 22:26:12,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:26:12,649::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107672 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:26:16,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:26:16,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163979 s, 250 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:26:21,414::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:26:21,415::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:26:21,857::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:26:21,864::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:26:21,866::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:26:22,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:26:22,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105728 s, 387 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:26:26,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:26:26,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.027078 s, 151 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:26:32,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:26:32,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077376 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:26:36,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:26:36,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823171 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:26:42,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:26:42,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087114 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:26:46,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:26:46,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000927432 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:26:52,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:26:52,936::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135985 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:26:56,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:26:56,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766208 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:27:02,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:27:03,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00098377 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:27:06,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:27:07,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791771 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:27:13,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:27:13,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839796 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:27:17,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:27:17,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156844 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14680::WARNING::2013-03-17 22:27:21,895::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:27:21,901::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 22:27:23,121::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 22:27:23,122::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 22:27:23,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:27:23,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0008561 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:27:23,152::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0008561 s, 2.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 22:27:23,153::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 22:27:23,153::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 22:27:23,156::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 22:27:23,157::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:27:23,158::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:27:23,159::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:27:23,159::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:27:23,160::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:27:23,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:27:23,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:27:23,587::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:27:23,588::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 22:27:23,590::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 22:27:23,591::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 22:27:23,592::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 22:27:23,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:27:23,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:27:23,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:27:24,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136536 s, 300 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:27:27,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:27:27,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768634 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:27:31,905::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:27:31,906::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:27:32,321::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:27:32,330::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:27:32,332::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:27:34,079::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:27:34,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176801 s, 232 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:27:37,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:27:37,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00356299 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:27:44,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:27:44,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839442 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:27:47,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:27:47,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149706 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:27:54,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:27:54,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160245 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:27:57,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:27:57,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156221 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:28:04,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:28:04,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101759 s, 403 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:28:07,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:28:07,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838963 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:28:14,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:28:14,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147142 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:28:17,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:28:17,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187469 s, 218 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:28:24,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:28:24,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00776516 s, 527 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:28:27,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:28:27,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157069 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14681::WARNING::2013-03-17 22:28:32,359::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:28:32,363::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 22:28:34,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:28:34,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080976 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:28:37,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:28:37,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00459498 s, 891 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:28:42,364::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 22:28:42,366::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:28:42,367::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:28:42,828::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:28:42,835::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:28:42,836::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:28:44,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:28:44,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114484 s, 358 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:28:47,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:28:47,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788686 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:28:54,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:28:54,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013842 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:28:57,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:28:57,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793784 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:29:04,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:29:04,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100116 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:29:07,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:29:07,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169591 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:29:14,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:29:14,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100638 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:29:17,508::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 22:29:17,509::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 22:29:17,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:29:17,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000837188 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:29:17,539::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000837188 s, 2.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 22:29:17,540::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 22:29:17,540::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 22:29:17,542::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 22:29:17,543::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:29:17,543::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:29:17,544::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:29:17,544::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:29:17,544::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:29:17,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:29:17,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:29:17,989::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:29:17,990::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 22:29:17,993::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 22:29:17,994::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 22:29:17,994::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 22:29:17,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:29:18,372::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:29:18,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:29:18,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0252861 s, 162 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:29:24,933::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:29:24,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00811098 s, 505 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:29:28,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:29:28,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135469 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:29:35,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:29:35,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747329 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:29:38,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:29:38,512::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078978 s, 5.2 MB/s\n'; <rc> = 0
>Thread-14682::WARNING::2013-03-17 22:29:42,869::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:29:42,873::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 22:29:45,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:29:45,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904836 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:29:48,522::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:29:48,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773085 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:29:52,884::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:29:52,885::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:29:53,361::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:29:53,368::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:29:53,369::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:29:55,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:29:55,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116023 s, 353 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:29:58,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:29:58,598::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143028 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:30:05,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:30:05,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848664 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:30:08,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:30:08,649::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151998 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:30:15,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:30:15,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838053 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:30:18,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:30:18,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100206 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:30:25,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:30:25,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825378 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:30:28,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:30:28,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807952 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:30:35,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:30:35,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932878 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:30:38,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:30:38,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0466578 s, 87.8 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:30:45,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:30:45,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142969 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:30:48,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:30:48,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000730499 s, 5.6 MB/s\n'; <rc> = 0
>Thread-14683::WARNING::2013-03-17 22:30:53,395::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:30:53,398::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 22:30:55,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:30:55,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139672 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:30:58,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:30:58,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806947 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:31:03,408::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:31:03,409::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:31:03,870::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:31:03,878::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:31:03,879::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:31:05,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:31:05,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127045 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:31:08,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:31:08,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0384692 s, 106 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:31:15,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:31:15,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872825 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:31:18,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:31:19,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845814 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:31:25,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:31:25,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176443 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:31:29,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:31:29,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128693 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:31:35,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:31:35,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075812 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:31:39,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:31:39,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149203 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:31:45,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:31:46,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146382 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:31:49,117::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:31:49,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825152 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:31:56,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:31:56,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001408 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:31:59,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:31:59,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149509 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14684::WARNING::2013-03-17 22:32:03,905::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:32:03,910::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 22:32:06,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:32:06,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016692 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:32:09,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:32:09,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0418856 s, 97.8 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:32:13,915::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:32:13,917::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:32:14,371::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:32:14,378::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:32:14,379::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:32:16,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:32:16,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155101 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:32:19,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:32:19,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824085 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:32:26,332::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 22:32:26,333::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 22:32:26,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:32:26,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00126964 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:32:26,365::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00126964 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 22:32:26,365::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 22:32:26,365::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 22:32:26,367::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 22:32:26,368::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:32:26,369::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:32:26,369::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:32:26,369::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:32:26,370::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:32:26,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:32:26,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:32:26,816::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:32:26,817::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 22:32:26,819::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 22:32:26,821::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 22:32:26,821::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 22:32:26,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:32:27,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:32:27,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:32:27,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167883 s, 244 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:32:29,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:32:29,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00576393 s, 711 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:32:37,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:32:37,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885513 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:32:39,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:32:39,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807115 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:32:47,416::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:32:47,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165505 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:32:49,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:32:49,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134264 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:32:57,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:32:57,521::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151021 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:32:59,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:32:59,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155048 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:33:07,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:33:07,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125866 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:33:09,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:33:09,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164859 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14685::WARNING::2013-03-17 22:33:14,407::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:33:14,411::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 22:33:17,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:33:17,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088056 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:33:19,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:33:19,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0320079 s, 128 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:33:24,417::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:33:24,418::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:33:24,940::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:33:24,947::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:33:24,961::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:33:27,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:33:27,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159421 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:33:29,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:33:29,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822502 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:33:37,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:33:37,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866937 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:33:39,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:33:39,692::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142171 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:33:47,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:33:47,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843024 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:33:49,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:33:49,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102037 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:33:57,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:33:57,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888681 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:33:59,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:33:59,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00230164 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:34:08,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:34:08,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875094 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:34:09,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:34:09,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798393 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:34:18,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:34:18,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00229035 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:34:19,817::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 22:34:19,818::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 22:34:19,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:34:19,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000814313 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:34:19,854::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000814313 s, 2.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 22:34:19,854::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 22:34:19,854::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 22:34:19,857::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 22:34:19,858::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:34:19,858::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:34:19,859::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:34:19,859::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:34:19,860::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:34:19,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:34:20,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:34:20,300::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:34:20,301::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 22:34:20,304::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 22:34:20,305::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 22:34:20,305::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 22:34:20,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:34:20,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:34:20,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:34:20,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174495 s, 235 kB/s\n'; <rc> = 0
>Thread-14686::WARNING::2013-03-17 22:34:25,008::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:34:25,018::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 22:34:28,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:34:28,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836925 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:34:30,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:34:30,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154246 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:34:35,021::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 22:34:35,022::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:34:35,023::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:34:35,495::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:34:35,502::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:34:35,503::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:34:38,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:34:38,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163437 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:34:40,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:34:40,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139352 s, 294 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:34:48,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:34:48,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860377 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:34:50,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:34:50,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131594 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:34:58,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:34:58,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158774 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:35:00,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:35:00,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138817 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:35:08,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:35:08,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000961978 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:35:10,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:35:10,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.002122 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:35:18,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:35:18,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102643 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:35:20,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:35:20,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821058 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:35:28,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:35:28,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107122 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:35:30,993::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:35:31,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137764 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14687::WARNING::2013-03-17 22:35:35,546::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:35:35,558::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 22:35:38,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:35:38,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00258688 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:35:41,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:35:41,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787703 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:35:45,562::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:35:45,563::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:35:45,983::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:35:45,989::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:35:45,993::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:35:48,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:35:48,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892928 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:35:51,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:35:51,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.030095 s, 136 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:35:58,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:35:58,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775902 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:36:01,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:36:01,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000965129 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:36:08,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:36:08,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781398 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:36:11,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:36:11,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102075 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:36:19,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:36:19,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118506 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:36:21,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:36:21,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000931069 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:36:29,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:36:29,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761934 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:36:31,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:36:31,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00320915 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:36:39,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:36:39,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123556 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:36:41,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:36:41,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817517 s, 5.0 MB/s\n'; <rc> = 0
>Thread-14688::WARNING::2013-03-17 22:36:46,023::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:36:46,029::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 22:36:49,227::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:36:49,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899055 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:36:51,341::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:36:51,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00759594 s, 539 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:36:56,032::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:36:56,033::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:36:56,472::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:36:56,478::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:36:56,480::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:36:59,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:36:59,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795464 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:37:01,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:37:01,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143762 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:37:09,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:37:09,409::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145826 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:37:11,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:37:11,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000677853 s, 6.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:37:19,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:37:19,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128812 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:37:21,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:37:21,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810508 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:37:29,520::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 22:37:29,521::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 22:37:29,522::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:37:29,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000684754 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:37:29,553::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000684754 s, 3.0 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 22:37:29,553::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 22:37:29,554::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 22:37:29,563::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 22:37:29,574::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:37:29,583::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:37:29,583::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:37:29,584::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:37:29,584::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:37:29,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:37:29,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:37:29,988::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:37:29,989::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 22:37:29,992::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 22:37:29,993::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 22:37:29,994::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 22:37:29,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:37:30,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:37:30,373::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:37:30,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125163 s, 327 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:37:31,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:37:31,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193094 s, 212 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:37:40,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:37:40,490::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737124 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:37:41,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:37:41,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170392 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:37:50,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:37:50,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00255765 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:37:51,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:37:51,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00263121 s, 1.6 MB/s\n'; <rc> = 0
>Thread-14689::WARNING::2013-03-17 22:37:56,510::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:37:56,516::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 22:38:00,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:38:00,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755448 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:38:01,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:38:01,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00203474 s, 2.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:38:06,518::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:38:06,520::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:38:06,919::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:38:06,926::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:38:06,927::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:38:10,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:38:10,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0198546 s, 206 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:38:11,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:38:11,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000719177 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:38:20,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:38:20,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084586 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:38:21,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:38:21,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824052 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:38:30,863::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:38:30,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077268 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:38:31,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:38:32,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129874 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:38:40,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:38:40,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845424 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:38:42,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:38:42,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000691604 s, 5.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:38:51,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:38:51,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143661 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:38:52,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:38:52,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761949 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:39:01,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:39:01,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010408 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:39:02,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:39:02,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746031 s, 5.5 MB/s\n'; <rc> = 0
>Thread-14690::WARNING::2013-03-17 22:39:06,976::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:39:06,980::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 22:39:11,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:39:11,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920523 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:39:12,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:39:12,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113592 s, 3.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:39:16,982::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:39:16,983::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:39:17,412::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:39:17,419::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:39:17,420::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:39:21,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:39:21,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841559 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:39:22,212::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 22:39:22,213::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 22:39:22,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:39:22,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00750462 s, 273 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:39:22,259::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00750462 s, 273 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 22:39:22,260::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 22:39:22,260::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 22:39:22,261::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 22:39:22,263::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:39:22,263::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:39:22,264::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:39:22,264::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:39:22,266::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:39:22,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:39:22,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:39:22,697::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:39:22,698::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 22:39:22,701::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 22:39:22,702::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 22:39:22,702::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 22:39:22,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:39:23,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:39:23,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:39:23,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00326097 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:39:31,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:39:31,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00215161 s, 1.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:39:33,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:39:33,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145158 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:39:41,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:39:41,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789896 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:39:43,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:39:43,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103261 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:39:51,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:39:51,484::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861999 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:39:53,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:39:53,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000921123 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:40:01,532::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:40:01,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150197 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:40:03,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:40:03,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130094 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:40:11,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:40:11,657::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152247 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:40:13,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:40:13,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138254 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14691::WARNING::2013-03-17 22:40:17,446::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:40:17,449::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 22:40:21,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:40:21,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104768 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:40:23,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:40:23,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00520632 s, 787 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:40:27,451::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 22:40:27,452::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:40:27,453::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:40:27,886::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:40:27,893::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:40:27,894::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:40:31,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:40:31,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169282 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:40:33,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:40:33,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791812 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:40:41,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:40:41,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151369 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:40:43,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:40:43,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000723767 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:40:51,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:40:51,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000933337 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:40:53,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:40:53,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776434 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:41:02,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:41:02,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014144 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:41:03,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:41:03,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159513 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:41:12,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:41:12,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141607 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:41:13,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:41:13,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166968 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:41:22,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:41:22,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123261 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:41:23,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:41:23,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0034054 s, 1.2 MB/s\n'; <rc> = 0
>Thread-14692::WARNING::2013-03-17 22:41:27,931::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:41:27,935::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 22:41:32,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:41:32,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000963133 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:41:33,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:41:33,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851914 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:41:37,943::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:41:37,944::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:41:38,378::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:41:38,385::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:41:38,386::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:41:42,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:41:42,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148884 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:41:43,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:41:43,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915102 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:41:52,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:41:52,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154582 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:41:53,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:41:53,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121685 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:42:02,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:42:02,526::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790942 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:42:03,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:42:03,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00373511 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:42:12,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:42:12,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000726572 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:42:13,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:42:13,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857376 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:42:22,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:42:22,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169831 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:42:23,781::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:42:23,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830932 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:42:32,721::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 22:42:32,723::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 22:42:32,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:42:32,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00128863 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:42:32,770::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00128863 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 22:42:32,771::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 22:42:32,771::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 22:42:32,774::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 22:42:32,776::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:42:32,776::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:42:32,777::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:42:32,777::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:42:32,778::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:42:32,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:42:33,222::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:42:33,227::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:42:33,228::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 22:42:33,230::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 22:42:33,231::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 22:42:33,232::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 22:42:33,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:42:33,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:42:33,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:42:33,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00822576 s, 498 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:42:33,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:42:33,880::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01774 s, 231 kB/s\n'; <rc> = 0
>Thread-14693::WARNING::2013-03-17 22:42:38,417::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:42:38,423::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 22:42:43,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:42:43,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00484599 s, 845 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:42:43,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:42:43,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782647 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:42:48,425::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:42:48,427::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:42:48,848::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:42:48,855::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:42:48,856::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:42:53,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:42:53,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155726 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:42:53,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:42:53,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170022 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:43:03,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:43:03,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000889336 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:43:03,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:43:03,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000729652 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:43:13,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:43:14,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:43:14,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010264 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:43:14,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109293 s, 375 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:43:24,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:43:24,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:43:24,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011337 s, 361 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:43:24,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741513 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:43:34,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:43:34,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104095 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:43:34,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:43:34,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101917 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:43:44,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:43:44,352::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:43:44,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134795 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:43:44,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116867 s, 350 kB/s\n'; <rc> = 0
>Thread-14694::WARNING::2013-03-17 22:43:48,903::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:43:48,905::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:43:54,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:43:54,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896302 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:43:54,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:43:54,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163321 s, 2.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:43:58,910::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:43:58,911::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:43:59,336::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:43:59,343::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:43:59,344::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:44:04,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:44:04,513::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134971 s, 303 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:44:04,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:44:04,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081864 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:44:14,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:44:14,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000922777 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:44:14,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:44:14,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761307 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:44:24,553::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 22:44:24,555::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 22:44:24,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:44:24,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0137516 s, 149 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:44:24,596::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0137516 s, 149 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 22:44:24,597::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 22:44:24,597::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 22:44:24,599::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 22:44:24,600::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:44:24,600::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:44:24,601::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:44:24,601::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:44:24,601::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:44:24,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:44:24,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:44:24,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00269357 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:44:25,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:44:25,047::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:44:25,048::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 22:44:25,050::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 22:44:25,051::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 22:44:25,052::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 22:44:25,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:44:25,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:44:25,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:44:25,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131129 s, 312 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:44:34,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:44:34,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147801 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:44:35,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:44:35,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015363 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:44:44,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:44:44,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158275 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:44:45,532::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:44:45,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154698 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:44:54,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:44:54,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134922 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:44:55,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:44:55,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799098 s, 5.1 MB/s\n'; <rc> = 0
>Thread-14695::WARNING::2013-03-17 22:44:59,381::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:44:59,387::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 22:45:05,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:45:05,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752559 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:45:05,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:45:05,658::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102053 s, 4.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:45:09,390::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:45:09,391::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:45:09,840::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:45:09,847::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:45:09,849::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:45:15,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:45:15,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00415055 s, 987 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:45:15,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:45:15,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168666 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:45:25,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:45:25,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844813 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:45:25,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:45:25,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00204468 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:45:35,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:45:35,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760066 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:45:35,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:45:35,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0027053 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:45:45,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:45:45,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901703 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:45:45,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:45:45,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00550306 s, 744 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:45:55,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:45:55,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162441 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:45:55,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:45:55,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814427 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:46:05,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:46:05,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149865 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:46:05,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:46:05,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149751 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14696::WARNING::2013-03-17 22:46:09,879::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:46:09,887::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 22:46:15,571::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:46:15,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919506 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:46:15,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:46:15,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871145 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:46:19,889::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 22:46:19,890::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:46:19,891::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:46:20,374::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:46:20,381::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:46:20,383::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:46:25,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:46:25,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088553 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:46:25,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:46:25,992::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792104 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:46:35,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:46:35,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807426 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:46:35,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:46:36,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149687 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:46:45,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:46:45,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857611 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:46:46,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:46:46,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846853 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:46:55,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:46:55,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170468 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:46:56,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:46:56,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148036 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:47:05,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:47:05,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814222 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:47:06,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:47:06,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151143 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:47:16,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:47:16,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127368 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:47:16,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:47:16,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00409756 s, 1000 kB/s\n'; <rc> = 0
>Thread-14697::WARNING::2013-03-17 22:47:20,428::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:47:20,433::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 22:47:26,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:47:26,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769376 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:47:26,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:47:26,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00187095 s, 2.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:47:30,435::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:47:30,437::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:47:30,857::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:47:30,864::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:47:30,865::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:47:36,151::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 22:47:36,154::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 22:47:36,156::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:47:36,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0247813 s, 82.6 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:47:36,207::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0247813 s, 82.6 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 22:47:36,208::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 22:47:36,208::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 22:47:36,209::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 22:47:36,210::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:47:36,210::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:47:36,211::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:47:36,211::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:47:36,212::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:47:36,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:47:36,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:47:36,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145744 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:47:36,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:47:36,657::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:47:36,658::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 22:47:36,662::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 22:47:36,662::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 22:47:36,663::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 22:47:36,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:47:37,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:47:37,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:47:37,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00724795 s, 565 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:47:46,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:47:46,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876005 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:47:47,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:47:47,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162012 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:47:56,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:47:56,409::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806846 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:47:57,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:47:57,238::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152786 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:48:06,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:48:06,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012804 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:48:07,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:48:07,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149774 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:48:16,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:48:16,490::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00070435 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:48:17,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:48:17,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00747401 s, 548 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:48:26,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:48:26,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158748 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:48:27,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:48:27,481::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126652 s, 3.2 MB/s\n'; <rc> = 0
>Thread-14698::WARNING::2013-03-17 22:48:30,889::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:48:30,893::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:48:36,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:48:36,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101438 s, 404 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:48:37,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:48:37,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149258 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:48:40,895::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:48:40,896::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:48:41,302::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:48:41,309::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:48:41,311::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:48:46,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:48:46,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000957725 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:48:47,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:48:47,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164674 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:48:56,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:48:56,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759159 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:48:57,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:48:57,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154437 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:49:06,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:49:06,692::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815044 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:49:07,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:49:07,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154341 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:49:16,697::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:49:16,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146647 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:49:17,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:49:17,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162766 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:49:26,744::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 22:49:26,745::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 22:49:26,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:49:26,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00096545 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:49:26,780::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00096545 s, 2.1 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 22:49:26,781::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 22:49:26,781::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 22:49:26,785::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 22:49:26,787::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:49:26,787::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:49:26,788::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:49:26,788::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:49:26,788::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:49:26,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:49:27,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:49:27,224::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:49:27,224::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 22:49:27,227::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 22:49:27,228::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 22:49:27,228::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 22:49:27,230::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:49:27,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:49:27,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:49:27,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.012884 s, 318 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:49:27,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:49:27,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01057 s, 388 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:49:37,647::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:49:37,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151334 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:49:38,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:49:38,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169822 s, 241 kB/s\n'; <rc> = 0
>Thread-14699::WARNING::2013-03-17 22:49:41,338::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:49:41,342::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:49:47,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:49:47,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000736349 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:49:48,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:49:48,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165313 s, 2.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:49:51,351::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:49:51,352::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:49:51,778::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:49:51,785::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:49:51,786::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:49:57,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:49:57,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834932 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:49:58,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:49:58,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000928282 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:50:07,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:50:07,793::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798011 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:50:08,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:50:08,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176323 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:50:17,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:50:17,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000721044 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:50:18,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:50:18,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865748 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:50:27,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:50:27,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869863 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:50:28,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:50:28,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00325589 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:50:37,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:50:37,907::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146075 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:50:38,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:50:38,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134254 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:50:47,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:50:47,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151696 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:50:48,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:50:48,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149366 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14700::WARNING::2013-03-17 22:50:51,817::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:50:51,824::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:50:57,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:50:58,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796265 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:50:58,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:50:58,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111554 s, 3.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:51:01,844::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:51:01,845::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:51:02,273::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:51:02,279::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:51:02,281::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:51:08,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:51:08,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174172 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:51:08,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:51:08,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123548 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:51:18,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:51:18,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143592 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:51:18,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:51:18,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765099 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:51:28,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:51:28,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838802 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:51:28,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:51:28,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765908 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:51:38,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:51:38,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138233 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:51:38,941::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:51:38,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175162 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:51:48,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:51:48,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165893 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:51:49,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:51:49,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859543 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:51:58,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:51:58,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760199 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:51:59,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:51:59,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128507 s, 3.2 MB/s\n'; <rc> = 0
>Thread-14701::WARNING::2013-03-17 22:52:02,308::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:52:02,312::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:52:08,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:52:08,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764636 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:52:09,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:52:09,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825846 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:52:12,321::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 22:52:12,322::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:52:12,323::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:52:12,762::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:52:12,768::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:52:12,770::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:52:18,320::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:52:18,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925403 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:52:19,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:52:19,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839384 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:52:28,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:52:28,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141948 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:52:29,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:52:29,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808743 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:52:38,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:52:38,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805023 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:52:39,377::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 22:52:39,378::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 22:52:39,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:52:39,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000856574 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:52:39,407::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000856574 s, 2.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 22:52:39,408::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 22:52:39,408::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 22:52:39,410::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 22:52:39,412::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:52:39,413::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:52:39,413::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:52:39,413::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:52:39,414::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:52:39,415::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:52:39,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:52:39,842::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:52:39,843::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 22:52:39,846::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 22:52:39,847::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 22:52:39,848::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 22:52:39,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:52:40,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:52:40,221::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:52:40,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0057865 s, 708 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:52:48,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:52:48,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743761 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:52:50,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:52:50,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148103 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:52:58,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:52:58,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160454 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:53:00,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:53:00,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175649 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:53:08,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:53:08,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126361 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:53:10,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:53:10,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150253 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14702::WARNING::2013-03-17 22:53:12,798::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:53:12,800::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:53:18,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:53:18,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861883 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:53:20,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:53:20,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764627 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:53:22,803::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:53:22,804::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:53:23,244::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:53:23,251::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:53:23,252::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:53:28,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:53:28,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741489 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:53:30,648::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:53:30,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149655 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:53:38,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:53:38,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153543 s, 267 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:53:40,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:53:40,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835813 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:53:48,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:53:48,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160519 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:53:50,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:53:50,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769503 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:53:58,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:53:58,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000701017 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:54:00,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:54:00,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166641 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:54:08,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:54:08,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00853574 s, 480 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:54:10,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:54:11,010::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147145 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:54:18,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:54:18,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100002 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:54:21,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:54:21,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112991 s, 3.6 MB/s\n'; <rc> = 0
>Thread-14703::WARNING::2013-03-17 22:54:23,302::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:54:23,304::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:54:28,912::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 22:54:28,913::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 22:54:28,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:54:28,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000747092 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:54:28,989::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000747092 s, 2.7 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 22:54:28,989::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 22:54:28,989::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 22:54:28,991::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 22:54:28,992::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:54:28,992::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:54:28,993::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:54:28,993::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:54:28,994::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:54:28,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:54:29,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:54:29,436::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:54:29,437::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 22:54:29,439::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 22:54:29,441::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 22:54:29,441::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 22:54:29,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:54:29,791::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:54:29,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:54:29,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754444 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:54:31,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:54:31,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00683294 s, 599 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:54:33,311::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:54:33,312::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:54:33,724::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:54:33,730::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:54:33,732::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:54:39,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:54:39,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919786 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:54:41,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:54:41,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876938 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:54:49,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:54:49,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854716 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:54:51,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:54:51,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081528 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:54:59,910::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:54:59,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115396 s, 355 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:55:01,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:55:01,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000943408 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:55:09,959::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:55:09,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00182705 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:55:11,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:55:11,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147213 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:55:19,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:55:20,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744262 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:55:21,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:55:21,562::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165028 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:55:30,048::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:55:30,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112645 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:55:31,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:55:31,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00331213 s, 1.2 MB/s\n'; <rc> = 0
>Thread-14704::WARNING::2013-03-17 22:55:33,762::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:55:33,767::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:55:40,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:55:40,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897622 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:55:41,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:55:41,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00460541 s, 889 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:55:43,769::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:55:43,770::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:55:44,286::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:55:44,293::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:55:44,294::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:55:50,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:55:50,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894215 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:55:51,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:55:51,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00183925 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:56:00,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:56:00,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744522 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:56:01,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:56:01,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00389056 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:56:10,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:56:10,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144102 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:56:11,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:56:11,979::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155798 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:56:20,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:56:20,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000939006 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:56:22,019::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:56:22,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00678228 s, 604 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:56:30,292::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:56:30,333::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138329 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:56:32,099::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:56:32,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799687 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:56:40,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:56:40,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140975 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:56:42,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:56:42,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111972 s, 3.7 MB/s\n'; <rc> = 0
>Thread-14705::WARNING::2013-03-17 22:56:44,338::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:56:44,345::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:56:50,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:56:50,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767468 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:56:52,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:56:52,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000739537 s, 5.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:56:54,353::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:56:54,354::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:56:54,748::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:56:54,755::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:56:54,756::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:57:00,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:57:00,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000943671 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:57:02,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:57:02,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148788 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:57:10,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:57:10,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895388 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:57:12,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:57:12,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00372976 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:57:20,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:57:20,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844252 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:57:22,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:57:22,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164581 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:57:30,532::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:57:30,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145102 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:57:32,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:57:32,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000928472 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:57:40,596::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:57:40,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794888 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:57:42,662::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 22:57:42,663::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 22:57:42,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:57:42,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0089901 s, 228 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:57:42,709::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0089901 s, 228 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 22:57:42,710::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 22:57:42,710::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 22:57:42,713::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 22:57:42,714::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:57:42,715::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:57:42,715::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:57:42,715::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:57:42,716::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 22:57:42,717::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:57:43,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:57:43,137::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 22:57:43,138::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 22:57:43,141::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 22:57:43,142::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 22:57:43,142::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 22:57:43,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:57:43,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:57:43,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:57:43,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206906 s, 198 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:57:50,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:57:50,667::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123045 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:57:53,730::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:57:53,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189199 s, 216 kB/s\n'; <rc> = 0
>Thread-14706::WARNING::2013-03-17 22:57:54,782::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:57:54,788::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:58:00,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:58:00,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206861 s, 198 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:58:03,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:58:03,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178724 s, 229 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:58:04,793::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 22:58:04,795::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:58:04,796::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:58:05,254::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:58:05,261::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:58:05,262::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:58:10,732::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:58:10,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799944 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:58:13,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:58:14,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821854 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:58:20,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:58:20,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859272 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:58:24,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:58:24,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0353491 s, 116 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:58:30,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:58:30,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851638 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:58:34,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:58:34,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148114 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:58:40,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:58:40,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00274371 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:58:44,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:58:44,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117641 s, 348 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:58:50,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:58:50,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151263 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:58:54,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:58:54,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014253 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:59:00,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:59:01,017::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131273 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:59:04,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:59:04,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168382 s, 2.4 MB/s\n'; <rc> = 0
>Thread-14707::WARNING::2013-03-17 22:59:05,314::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 22:59:05,320::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 22:59:11,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:59:11,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844347 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:59:15,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:59:15,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076901 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 22:59:15,323::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 22:59:15,324::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 22:59:15,731::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 22:59:15,738::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 22:59:15,739::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:59:21,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:59:21,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000985415 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:59:25,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:59:25,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786029 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:59:31,101::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 22:59:31,102::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 22:59:31,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:59:31,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000717157 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:59:31,133::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000717157 s, 2.9 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 22:59:31,133::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 22:59:31,134::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 22:59:31,135::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 22:59:31,136::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:59:31,137::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:59:31,137::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:59:31,138::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:59:31,138::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 22:59:31,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:59:31,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:59:31,567::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 22:59:31,568::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 22:59:31,570::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 22:59:31,572::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 22:59:31,572::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 22:59:31,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:59:31,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:59:31,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:59:32,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00783608 s, 523 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:59:35,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:59:35,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872116 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:59:42,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:59:42,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839342 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:59:45,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:59:45,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014352 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 22:59:52,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 22:59:52,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850678 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 22:59:55,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 22:59:55,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116452 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:00:02,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:00:02,270::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.04448 s, 92.1 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:00:05,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:00:05,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143374 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:00:12,280::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:00:12,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169499 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:00:15,492::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:00:15,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773513 s, 5.3 MB/s\n'; <rc> = 0
>Thread-14708::WARNING::2013-03-17 23:00:15,772::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:00:15,782::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 23:00:22,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:00:22,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841677 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:00:25,796::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:00:25,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:00:25,805::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:00:25,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135146 s, 3.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:00:26,296::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:00:26,301::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:00:26,303::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:00:32,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:00:32,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133155 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:00:35,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:00:35,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130732 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:00:42,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:00:42,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00449679 s, 911 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:00:46,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:00:46,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774969 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:00:52,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:00:52,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131466 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:00:56,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:00:56,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139422 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:01:02,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:01:02,591::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122863 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:01:06,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:01:06,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015708 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:01:12,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:01:12,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000727524 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:01:16,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:01:16,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120036 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:01:22,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:01:22,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013078 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14709::WARNING::2013-03-17 23:01:26,334::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:01:26,338::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:01:26,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:01:26,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162335 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:01:32,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:01:32,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000914124 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:01:36,340::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:01:36,341::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:01:36,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:01:36,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00781737 s, 524 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:01:36,826::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:01:36,831::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:01:36,833::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:01:42,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:01:42,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855811 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:01:46,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:01:46,548::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000988109 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:01:52,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:01:52,809::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148731 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:01:56,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:01:56,617::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880464 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:02:02,814::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:02:02,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150197 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:02:06,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:02:06,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000986569 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:02:12,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:02:12,881::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00370541 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:02:16,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:02:16,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000960462 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:02:22,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:02:22,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783109 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:02:26,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:02:26,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152053 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:02:32,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:02:32,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00535419 s, 765 kB/s\n'; <rc> = 0
>Thread-14710::WARNING::2013-03-17 23:02:36,857::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:02:36,861::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:02:36,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:02:36,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906161 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:02:42,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:02:43,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757702 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:02:46,864::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:02:46,865::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:02:46,955::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 23:02:46,958::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 23:02:46,959::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:02:47,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00144913 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:02:47,009::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00144913 s, 1.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 23:02:47,010::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 23:02:47,010::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 23:02:47,012::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 23:02:47,013::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm reload operation' is holding the operation mutex, waiting...
>Thread-21::DEBUG::2013-03-17 23:02:47,342::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:02:47,347::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:02:47,348::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:02:47,358::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:02:47,358::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:02:47,359::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:02:47,359::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:02:47,360::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:02:47,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:02:47,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:02:47,763::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:02:47,764::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 23:02:47,767::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 23:02:47,768::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 23:02:47,768::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 23:02:47,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:02:48,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:02:48,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:02:48,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00938008 s, 437 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:02:53,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:02:53,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915587 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:02:58,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:02:58,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892289 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:03:03,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:03:03,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0020058 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:03:08,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:03:08,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075804 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:03:13,092::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:03:13,130::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139081 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:03:18,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:03:18,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013615 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:03:23,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:03:23,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106726 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:03:28,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:03:28,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791502 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:03:33,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:03:33,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0213476 s, 192 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:03:38,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:03:38,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141997 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:03:43,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:03:43,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143796 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14711::WARNING::2013-03-17 23:03:47,418::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:03:47,423::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:03:48,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:03:48,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740132 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:03:53,281::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:03:53,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000926721 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:03:57,425::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 23:03:57,427::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:03:57,428::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:03:57,845::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:03:57,852::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:03:57,853::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:03:58,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:03:58,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00934907 s, 438 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:04:03,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:04:03,358::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165355 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:04:08,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:04:08,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812837 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:04:13,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:04:13,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016311 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:04:18,797::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:04:18,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000908239 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:04:23,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:04:23,456::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137458 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:04:28,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:04:28,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868688 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:04:33,460::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 23:04:33,461::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 23:04:33,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:04:33,490::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000844997 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:04:33,491::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000844997 s, 2.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 23:04:33,492::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 23:04:33,492::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 23:04:33,494::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 23:04:33,495::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:04:33,495::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:04:33,496::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:04:33,496::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:04:33,497::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:04:33,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:04:33,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:04:33,932::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:04:33,933::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 23:04:33,935::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 23:04:33,936::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 23:04:33,937::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 23:04:33,938::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:04:34,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:04:34,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:04:34,402::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133333 s, 307 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:04:38,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:04:39,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0366232 s, 112 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:04:44,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:04:44,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010899 s, 376 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:04:49,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:04:49,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164334 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:04:54,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:04:54,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011417 s, 359 kB/s\n'; <rc> = 0
>Thread-14712::WARNING::2013-03-17 23:04:57,883::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:04:57,886::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:04:59,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:04:59,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856348 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:05:04,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:05:04,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000930202 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:05:07,888::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:05:07,889::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:05:08,312::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:05:08,318::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:05:08,320::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:05:09,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:05:09,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00695773 s, 589 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:05:14,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:05:14,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00735574 s, 557 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:05:19,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:05:19,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159954 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:05:24,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:05:24,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762496 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:05:29,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:05:29,404::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000974496 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:05:34,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:05:34,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820428 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:05:39,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:05:39,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145642 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:05:44,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:05:44,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846759 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:05:49,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:05:49,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000979108 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:05:54,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:05:54,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798203 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:05:59,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:05:59,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106952 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:06:04,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:06:04,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147164 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14713::WARNING::2013-03-17 23:06:08,355::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:06:08,360::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:06:09,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:06:09,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00209511 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:06:14,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:06:14,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00451156 s, 908 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:06:18,365::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:06:18,367::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:06:18,812::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:06:18,819::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:06:18,820::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:06:19,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:06:19,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00767171 s, 534 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:06:24,857::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:06:24,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165412 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:06:29,814::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:06:29,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000714978 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:06:34,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:06:34,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000956931 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:06:39,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:06:39,907::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852561 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:06:44,941::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:06:44,979::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856726 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:06:49,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:06:49,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00730355 s, 561 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:06:54,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:06:55,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0285949 s, 143 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:07:00,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:07:00,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832057 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:07:05,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:07:05,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161917 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:07:10,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:07:10,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152943 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:07:15,108::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:07:15,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124151 s, 3.3 MB/s\n'; <rc> = 0
>Thread-14714::WARNING::2013-03-17 23:07:18,847::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:07:18,850::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:07:20,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:07:20,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883073 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:07:25,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:07:25,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907383 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:07:28,854::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:07:28,856::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:07:29,258::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:07:29,271::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:07:29,272::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:07:30,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:07:30,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0526538 s, 77.8 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:07:35,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:07:35,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830871 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:07:40,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:07:40,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144007 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:07:45,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:07:45,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798263 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:07:50,451::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 23:07:50,452::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 23:07:50,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:07:50,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000797812 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:07:50,483::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000797812 s, 2.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 23:07:50,484::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 23:07:50,484::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 23:07:50,485::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 23:07:50,487::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:07:50,487::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:07:50,488::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:07:50,488::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:07:50,488::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:07:50,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:07:50,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:07:50,917::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:07:50,918::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 23:07:50,921::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 23:07:50,922::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 23:07:50,923::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 23:07:50,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:07:51,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:07:51,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:07:51,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146483 s, 280 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:07:55,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:07:55,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137974 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:08:01,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:08:01,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104172 s, 393 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:08:05,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:08:05,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789124 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:08:11,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:08:11,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143186 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:08:15,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:08:15,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156782 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:08:21,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:08:21,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113249 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:08:25,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:08:25,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168789 s, 2.4 MB/s\n'; <rc> = 0
>Thread-14715::WARNING::2013-03-17 23:08:29,297::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:08:29,301::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:08:31,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:08:31,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797951 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:08:35,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:08:35,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835526 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:08:39,315::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:08:39,316::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:08:39,824::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:08:39,832::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:08:39,833::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:08:41,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:08:41,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124975 s, 328 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:08:45,465::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:08:45,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836964 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:08:51,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:08:51,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169886 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:08:55,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:08:55,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148563 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:09:01,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:09:01,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744215 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:09:05,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:09:05,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806048 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:09:11,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:09:12,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808129 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:09:15,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:09:15,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153492 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:09:22,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:09:22,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805699 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:09:25,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:09:25,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799991 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:09:32,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:09:32,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148258 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:09:35,685::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 23:09:35,687::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 23:09:35,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:09:35,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00215959 s, 948 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:09:35,725::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00215959 s, 948 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 23:09:35,726::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 23:09:35,727::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 23:09:35,728::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 23:09:35,729::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:09:35,730::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:09:35,730::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:09:35,731::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:09:35,731::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:09:35,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:09:36,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:09:36,163::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:09:36,164::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 23:09:36,166::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 23:09:36,168::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 23:09:36,168::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 23:09:36,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:09:36,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:09:36,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:09:36,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0254115 s, 161 kB/s\n'; <rc> = 0
>Thread-14716::WARNING::2013-03-17 23:09:39,860::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:09:39,866::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:09:42,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:09:42,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864147 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:09:46,612::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:09:46,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924584 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:09:49,868::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 23:09:49,870::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:09:49,871::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:09:50,414::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:09:50,421::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:09:50,422::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:09:52,324::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:09:52,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00711486 s, 576 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:09:56,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:09:56,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844611 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:10:02,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:10:02,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881791 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:10:06,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:10:06,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000967033 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:10:12,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:10:12,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771285 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:10:16,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:10:16,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012635 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:10:22,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:10:22,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016705 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:10:26,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:10:26,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139766 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:10:32,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:10:32,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000675304 s, 6.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:10:36,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:10:36,853::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134418 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:10:42,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:10:42,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148256 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:10:46,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:10:46,890::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000700358 s, 5.8 MB/s\n'; <rc> = 0
>Thread-14717::WARNING::2013-03-17 23:10:50,457::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:10:50,460::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:10:52,814::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:10:52,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813021 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:10:56,899::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:10:56,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830628 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:11:00,462::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:11:00,463::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:11:00,882::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:11:00,888::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:11:00,890::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:11:02,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:11:02,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113923 s, 360 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:11:06,933::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:11:06,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793698 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:11:12,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:11:12,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762609 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:11:16,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:11:17,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770712 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:11:23,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:11:23,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000944383 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:11:27,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:11:27,046::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106589 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:11:33,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:11:33,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141382 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:11:37,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:11:37,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147847 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:11:43,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:11:43,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078017 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:11:47,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:11:47,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133951 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:11:53,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:11:53,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016543 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:11:57,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:11:57,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873504 s, 4.7 MB/s\n'; <rc> = 0
>Thread-14718::WARNING::2013-03-17 23:12:00,915::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:12:00,919::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:12:03,345::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:12:03,372::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848556 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:12:07,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:12:07,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000964459 s, 4.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:12:10,921::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:12:10,922::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:12:11,333::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:12:11,340::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:12:11,342::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:12:13,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:12:13,478::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01526 s, 268 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:12:17,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:12:17,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142584 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:12:23,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:12:23,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145072 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:12:27,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:12:27,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144911 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:12:33,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:12:33,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00232163 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:12:37,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:12:37,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080403 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:12:43,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:12:43,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141302 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:12:47,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:12:47,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179305 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:12:53,754::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 23:12:53,759::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 23:12:53,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:12:53,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0015681 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:12:53,798::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0015681 s, 1.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 23:12:53,799::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 23:12:53,799::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 23:12:53,800::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 23:12:53,801::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:12:53,802::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:12:53,803::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:12:53,803::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:12:53,803::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:12:53,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:12:54,236::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:12:54,241::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:12:54,242::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 23:12:54,244::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 23:12:54,245::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 23:12:54,246::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 23:12:54,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:12:54,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:12:54,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:12:54,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0236526 s, 173 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:12:57,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:12:57,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168101 s, 244 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:13:04,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:13:04,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000946056 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:13:07,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:13:07,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139952 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14719::WARNING::2013-03-17 23:13:11,368::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:13:11,371::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:13:14,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:13:14,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075821 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:13:17,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:13:17,529::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154077 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:13:21,374::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:13:21,375::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:13:21,781::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:13:21,788::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:13:21,789::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:13:24,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:13:25,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0401659 s, 102 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:13:27,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:13:27,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015411 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:13:35,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:13:35,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128072 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:13:37,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:13:37,607::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835719 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:13:45,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:13:45,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144388 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:13:47,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:13:47,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00190303 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:13:55,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:13:55,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112326 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:13:57,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:13:57,698::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154144 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:14:05,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:14:05,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150289 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:14:07,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:14:07,733::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859686 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:14:15,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:14:15,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017109 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:14:17,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:14:17,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123026 s, 3.3 MB/s\n'; <rc> = 0
>Thread-14720::WARNING::2013-03-17 23:14:21,815::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:14:21,819::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:14:25,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:14:25,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865154 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:14:27,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:14:27,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869157 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:14:31,821::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:14:31,823::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:14:32,241::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:14:32,248::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:14:32,249::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:14:35,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:14:35,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104709 s, 391 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:14:37,823::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 23:14:37,824::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 23:14:37,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:14:37,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000948817 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:14:37,854::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000948817 s, 2.2 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 23:14:37,855::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 23:14:37,855::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 23:14:37,857::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 23:14:37,859::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:14:37,859::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:14:37,860::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:14:37,860::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:14:37,861::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:14:37,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:14:38,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:14:38,305::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:14:38,306::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 23:14:38,309::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 23:14:38,310::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 23:14:38,311::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 23:14:38,312::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:14:38,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:14:38,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:14:38,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139737 s, 293 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:14:45,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:14:45,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809527 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:14:48,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:14:48,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920326 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:14:55,797::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:14:55,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804085 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:14:58,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:14:58,881::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833211 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:15:05,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:15:05,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871396 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:15:08,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:15:08,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155003 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:15:15,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:15:15,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0208222 s, 197 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:15:18,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:15:18,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154237 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:15:26,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:15:26,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00180018 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:15:28,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:15:29,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015414 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14721::WARNING::2013-03-17 23:15:32,276::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:15:32,280::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:15:36,108::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:15:36,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781656 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:15:39,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:15:39,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790701 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:15:42,290::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 23:15:42,291::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:15:42,292::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:15:42,727::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:15:42,734::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:15:42,735::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:15:46,178::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:15:46,219::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113081 s, 362 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:15:49,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:15:49,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076254 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:15:56,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:15:56,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791907 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:15:59,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:15:59,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000974924 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:16:06,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:16:06,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000957859 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:16:09,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:16:09,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747522 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:16:16,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:16:16,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000718997 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:16:19,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:16:19,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000716594 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:16:26,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:16:26,517::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00254292 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:16:29,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:16:29,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168411 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:16:36,556::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:16:36,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840132 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:16:39,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:16:39,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178275 s, 2.3 MB/s\n'; <rc> = 0
>Thread-14722::WARNING::2013-03-17 23:16:42,761::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:16:42,764::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:16:46,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:16:46,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867027 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:16:49,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:16:49,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871524 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:16:52,769::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:16:52,770::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:16:53,342::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:16:53,349::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:16:53,351::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:16:56,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:16:56,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143905 s, 285 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:16:59,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:16:59,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782356 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:17:06,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:17:06,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153003 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:17:09,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:17:09,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153372 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:17:16,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:17:16,880::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821342 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:17:19,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:17:19,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166457 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:17:26,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:17:26,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085991 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:17:29,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:17:29,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793688 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:17:36,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:17:37,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157019 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:17:39,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:17:39,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142942 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:17:47,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:17:47,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817244 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:17:49,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:17:49,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156126 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14723::WARNING::2013-03-17 23:17:53,383::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:17:53,389::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:17:57,147::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 23:17:57,149::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 23:17:57,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:17:57,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000779379 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:17:57,175::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000779379 s, 2.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 23:17:57,175::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 23:17:57,175::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 23:17:57,177::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 23:17:57,178::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:17:57,179::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:17:57,179::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:17:57,180::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:17:57,180::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:17:57,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:17:57,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:17:57,597::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:17:57,598::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 23:17:57,601::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 23:17:57,602::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 23:17:57,602::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 23:17:57,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:17:57,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:17:57,959::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:17:57,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100677 s, 407 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:17:59,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:17:59,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149216 s, 275 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:18:03,392::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:18:03,394::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:18:03,863::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:18:03,870::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:18:03,871::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:18:08,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:18:08,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159576 s, 257 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:18:09,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:18:09,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015799 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:18:18,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:18:18,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.021423 s, 191 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:18:19,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:18:19,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871767 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:18:28,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:18:28,311::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179841 s, 228 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:18:29,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:18:29,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795738 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:18:38,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:18:38,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000724626 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:18:39,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:18:39,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136467 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:18:48,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:18:48,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146877 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:18:49,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:18:49,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817556 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:18:58,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:18:58,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140575 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:18:59,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:18:59,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0021446 s, 1.9 MB/s\n'; <rc> = 0
>Thread-14724::WARNING::2013-03-17 23:19:03,900::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:19:03,903::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:19:08,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:19:08,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778943 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:19:09,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:19:09,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109269 s, 3.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:19:13,905::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:19:13,906::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:19:14,312::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:19:14,319::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:19:14,320::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:19:18,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:19:18,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178342 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:19:19,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:19:19,973::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165262 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:19:28,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:19:28,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795404 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:19:29,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:19:30,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00205428 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:19:38,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:19:38,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854309 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:19:40,015::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 23:19:40,017::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 23:19:40,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:19:40,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000777856 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:19:40,049::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000777856 s, 2.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 23:19:40,050::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 23:19:40,050::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 23:19:40,052::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 23:19:40,054::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:19:40,055::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:19:40,055::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:19:40,056::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:19:40,056::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:19:40,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:19:40,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:19:40,478::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:19:40,479::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 23:19:40,481::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 23:19:40,491::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 23:19:40,492::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 23:19:40,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:19:40,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:19:40,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:19:40,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163184 s, 251 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:19:48,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:19:48,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139656 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:19:50,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:19:50,950::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00070861 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:19:58,993::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:19:59,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082773 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:20:00,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:20:00,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0009115 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:20:09,060::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:20:09,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140134 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:20:10,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:20:11,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137674 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14725::WARNING::2013-03-17 23:20:14,347::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:20:14,351::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:20:19,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:20:19,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778402 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:20:21,045::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:20:21,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827779 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:20:24,353::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:20:24,354::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:20:24,776::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:20:24,783::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:20:24,784::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:20:29,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:20:29,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826866 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:20:31,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:20:31,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852842 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:20:39,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:20:39,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015048 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:20:41,118::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:20:41,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000926447 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:20:49,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:20:49,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00195887 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:20:51,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:20:51,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786247 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:20:59,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:20:59,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135037 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:21:01,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:21:01,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167726 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:21:09,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:21:09,569::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000734979 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:21:11,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:21:11,287::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134978 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:21:19,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:21:19,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788137 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:21:21,292::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:21:21,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124995 s, 3.3 MB/s\n'; <rc> = 0
>Thread-14726::WARNING::2013-03-17 23:21:24,809::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:21:24,812::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:21:29,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:21:29,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784969 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:21:31,345::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:21:31,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171421 s, 2.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:21:34,814::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 23:21:34,815::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:21:34,818::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:21:35,243::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:21:35,249::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:21:35,250::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:21:39,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:21:39,805::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842375 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:21:41,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:21:41,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000957269 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:21:49,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:21:49,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822737 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:21:51,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:21:51,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794961 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:21:59,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:21:59,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820206 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:22:01,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:22:01,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859789 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:22:09,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:22:10,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763969 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:22:11,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:22:11,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078792 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:22:20,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:22:20,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163353 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:22:21,605::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:22:21,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00385001 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:22:30,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:22:30,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126117 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:22:31,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:22:31,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0318509 s, 129 kB/s\n'; <rc> = 0
>Thread-14727::WARNING::2013-03-17 23:22:35,283::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:22:35,288::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:22:40,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:22:40,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919695 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:22:41,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:22:41,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00198298 s, 2.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:22:45,290::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:22:45,292::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:22:45,734::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:22:45,743::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:22:45,744::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:22:50,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:22:50,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890776 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:22:51,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:22:51,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740164 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:23:00,354::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 23:23:00,355::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 23:23:00,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:23:00,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00131419 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:23:00,392::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00131419 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 23:23:00,393::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 23:23:00,393::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 23:23:00,395::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 23:23:00,396::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:23:00,397::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:23:00,397::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:23:00,397::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:23:00,398::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:23:00,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:23:00,819::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:23:00,824::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:23:00,825::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 23:23:00,828::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 23:23:00,829::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 23:23:00,830::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 23:23:00,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:23:01,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:23:01,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:23:01,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175337 s, 234 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:23:01,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:23:01,866::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189402 s, 216 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:23:11,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:23:11,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815391 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:23:11,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:23:11,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00206072 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:23:21,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:23:21,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774251 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:23:21,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:23:21,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163579 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:23:31,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:23:31,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014008 s, 292 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:23:31,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:23:32,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137064 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:23:41,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:23:41,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153329 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:23:42,019::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:23:42,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141175 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14728::WARNING::2013-03-17 23:23:45,776::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:23:45,780::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:23:51,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:23:51,692::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884894 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:23:52,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:23:52,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761749 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:23:55,782::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:23:55,783::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:23:56,198::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:23:56,204::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:23:56,206::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:24:01,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:24:01,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126786 s, 323 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:24:02,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:24:02,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00259054 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:24:11,993::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:24:12,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000981263 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:24:12,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:24:12,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132825 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:24:22,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:24:22,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151374 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:24:22,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:24:22,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139863 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:24:32,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:24:32,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891889 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:24:32,254::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:24:32,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148467 s, 276 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:24:42,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:24:42,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743742 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:24:42,320::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 23:24:42,322::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 23:24:42,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:24:42,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0014561 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:24:42,362::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0014561 s, 1.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 23:24:42,362::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 23:24:42,363::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 23:24:42,364::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 23:24:42,365::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:24:42,366::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:24:42,366::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:24:42,367::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:24:42,367::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:24:42,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:24:42,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:24:42,807::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:24:42,808::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 23:24:42,810::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 23:24:42,811::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 23:24:42,812::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 23:24:42,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:24:43,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:24:43,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:24:43,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150537 s, 272 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:24:52,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:24:52,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013564 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:24:53,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:24:53,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080268 s, 5.1 MB/s\n'; <rc> = 0
>Thread-14729::WARNING::2013-03-17 23:24:56,234::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:24:56,237::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:25:02,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:25:02,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102184 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:25:03,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:25:03,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170988 s, 2.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:25:06,242::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:25:06,244::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:25:06,667::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:25:06,673::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:25:06,675::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:25:12,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:25:12,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881257 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:25:13,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:25:13,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014316 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:25:22,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:25:22,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787904 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:25:23,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:25:23,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753027 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:25:32,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:25:32,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802068 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:25:33,395::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:25:33,439::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156447 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:25:42,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:25:42,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154894 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:25:43,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:25:43,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084518 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:25:52,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:25:52,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000717175 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:25:53,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:25:53,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150086 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:26:02,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:26:02,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805229 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:26:03,537::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:26:03,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000991981 s, 4.1 MB/s\n'; <rc> = 0
>Thread-14730::WARNING::2013-03-17 23:26:06,708::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:26:06,715::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:26:12,894::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:26:12,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839917 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:26:13,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:26:13,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925845 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:26:16,718::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:26:16,719::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:26:17,149::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:26:17,156::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:26:17,157::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:26:22,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:26:22,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811205 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:26:23,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:26:23,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000958467 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:26:33,034::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:26:33,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152179 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:26:33,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:26:33,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153554 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:26:43,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:26:43,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000942839 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:26:43,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:26:43,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747434 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:26:53,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:26:53,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139414 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:26:53,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:26:53,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136684 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:27:03,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:27:03,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138907 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:27:03,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:27:03,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00304735 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:27:13,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:27:13,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154956 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:27:13,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:27:13,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140684 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14731::WARNING::2013-03-17 23:27:17,185::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:27:17,189::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:27:23,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:27:23,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788087 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:27:23,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:27:23,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765716 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:27:27,204::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 23:27:27,205::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:27:27,207::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:27:27,637::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:27:27,643::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:27:27,645::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:27:33,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:27:33,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743939 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:27:33,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:27:33,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000928439 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:27:43,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:27:43,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837047 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:27:43,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:27:43,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00223675 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:27:53,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:27:53,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122076 s, 336 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:27:54,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:27:54,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145052 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:28:03,730::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 23:28:03,732::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 23:28:03,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:28:03,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000778324 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:28:03,761::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000778324 s, 2.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 23:28:03,762::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 23:28:03,762::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 23:28:03,763::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 23:28:03,765::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:28:03,765::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:28:03,766::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:28:03,766::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:28:03,766::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:28:03,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:28:04,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:28:04,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111648 s, 367 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:28:04,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:28:04,235::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:28:04,236::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 23:28:04,238::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 23:28:04,239::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 23:28:04,240::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 23:28:04,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:28:04,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:28:04,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:28:04,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101098 s, 405 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:28:14,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:28:14,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137977 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:28:14,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:28:14,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145081 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:28:24,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:28:24,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00983125 s, 417 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:28:24,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:28:24,818::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124893 s, 3.3 MB/s\n'; <rc> = 0
>Thread-14732::WARNING::2013-03-17 23:28:27,672::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:28:27,676::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 23:28:34,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:28:34,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819195 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:28:34,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:28:34,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00964951 s, 424 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:28:37,678::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:28:37,680::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:28:38,105::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:28:38,113::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:28:38,115::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:28:44,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:28:44,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117499 s, 349 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:28:44,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:28:44,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102407 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:28:54,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:28:54,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145204 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:28:55,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:28:55,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165996 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:29:04,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:29:04,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841911 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:29:05,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:29:05,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00184571 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:29:14,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:29:14,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134042 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:29:15,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:29:15,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016055 s, 255 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:29:24,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:29:24,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150338 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:29:25,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:29:25,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883943 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:29:34,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:29:34,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169055 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:29:35,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:29:35,395::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128191 s, 320 kB/s\n'; <rc> = 0
>Thread-14733::WARNING::2013-03-17 23:29:38,141::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:29:38,145::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 23:29:44,564::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 23:29:44,565::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 23:29:44,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:29:44,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000890379 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:29:44,595::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000890379 s, 2.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 23:29:44,596::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 23:29:44,596::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 23:29:44,598::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 23:29:44,599::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:29:44,599::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:29:44,600::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:29:44,600::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:29:44,600::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:29:44,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:29:44,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:29:44,998::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:29:44,999::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 23:29:45,002::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 23:29:45,003::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 23:29:45,003::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 23:29:45,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:29:45,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:29:45,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:29:45,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101683 s, 403 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:29:45,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:29:45,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152012 s, 269 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:29:48,147::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:29:48,148::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:29:48,573::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:29:48,579::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:29:48,581::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:29:55,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:29:55,456::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100967 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:29:55,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:29:55,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146778 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:30:05,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:30:05,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148302 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:30:05,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:30:05,627::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851827 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:30:15,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:30:15,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796326 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:30:15,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:30:15,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152208 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:30:25,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:30:25,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141915 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:30:25,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:30:25,791::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145547 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:30:35,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:30:35,633::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130683 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:30:35,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:30:35,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820299 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:30:45,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:30:45,671::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000997028 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:30:45,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:30:45,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169168 s, 2.4 MB/s\n'; <rc> = 0
>Thread-14734::WARNING::2013-03-17 23:30:48,609::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:30:48,613::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 23:30:55,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:30:55,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179359 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:30:55,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:30:56,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846178 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:30:58,615::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:30:58,616::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:30:59,036::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:30:59,042::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:30:59,044::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:31:05,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:31:05,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139413 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:31:06,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:31:06,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809744 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:31:15,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:31:15,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821812 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:31:16,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:31:16,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824551 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:31:25,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:31:25,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000889324 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:31:26,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:31:26,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119918 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:31:35,857::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:31:35,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150964 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:31:36,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:31:36,331::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172936 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:31:45,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:31:45,958::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831502 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:31:46,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:31:46,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013326 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:31:55,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:31:56,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00225365 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:31:56,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:31:56,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088638 s, 4.6 MB/s\n'; <rc> = 0
>Thread-14735::WARNING::2013-03-17 23:31:59,073::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:31:59,076::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 23:32:06,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:32:06,046::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890565 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:32:06,530::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:32:06,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817061 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:32:09,078::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:32:09,079::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:32:09,561::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:32:09,567::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:32:09,569::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:32:16,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:32:16,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782115 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:32:16,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:32:16,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916067 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:32:26,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:32:26,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840001 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:32:26,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:32:26,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172414 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:32:36,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:32:36,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777124 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:32:36,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:32:36,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822267 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:32:46,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:32:46,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133488 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:32:46,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:32:46,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839388 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:32:56,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:32:56,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150671 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:32:56,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:32:56,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773093 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:33:06,254::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:33:06,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148789 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:33:06,977::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 23:33:06,979::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 23:33:06,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:33:07,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00140903 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:33:07,032::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00140903 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 23:33:07,033::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 23:33:07,033::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 23:33:07,035::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 23:33:07,037::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:33:07,038::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:33:07,039::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:33:07,040::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:33:07,040::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:33:07,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:33:07,478::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:33:07,483::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:33:07,484::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 23:33:07,486::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 23:33:07,488::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 23:33:07,488::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 23:33:07,490::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:33:07,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:33:08,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:33:08,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0371304 s, 110 kB/s\n'; <rc> = 0
>Thread-14736::WARNING::2013-03-17 23:33:09,598::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:33:09,604::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 23:33:16,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:33:16,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146446 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:33:18,481::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:33:18,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00395997 s, 1.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:33:19,607::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 23:33:19,608::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:33:19,609::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:33:20,026::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:33:20,033::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:33:20,035::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:33:26,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:33:26,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852594 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:33:28,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:33:28,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105599 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:33:36,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:33:36,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171653 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:33:38,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:33:38,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133159 s, 308 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:33:46,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:33:46,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148017 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:33:48,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:33:48,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135789 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:33:56,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:33:56,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136596 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:33:58,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:33:58,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185441 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:34:06,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:34:06,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832152 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:34:08,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:34:08,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144497 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:34:16,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:34:16,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899763 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:34:18,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:34:19,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0037351 s, 1.1 MB/s\n'; <rc> = 0
>Thread-14737::WARNING::2013-03-17 23:34:20,062::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:34:20,067::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 23:34:26,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:34:26,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824042 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:34:29,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:34:29,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102005 s, 402 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:34:30,071::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:34:30,073::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:34:30,478::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:34:30,484::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:34:30,485::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:34:36,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:34:36,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075932 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:34:39,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:34:39,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777052 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:34:46,685::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 23:34:46,686::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 23:34:46,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:34:46,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000832592 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:34:46,721::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000832592 s, 2.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 23:34:46,721::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 23:34:46,722::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 23:34:46,724::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 23:34:46,725::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:34:46,725::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:34:46,726::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:34:46,726::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:34:46,726::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:34:46,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:34:47,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:34:47,148::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:34:47,149::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 23:34:47,151::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 23:34:47,153::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 23:34:47,153::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 23:34:47,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:34:47,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:34:47,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:34:47,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140659 s, 291 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:34:49,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:34:49,791::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153663 s, 267 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:34:57,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:34:57,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142334 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:34:59,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:34:59,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0232919 s, 176 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:35:07,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:35:07,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126631 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:35:09,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:35:09,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160326 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:35:17,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:35:17,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793133 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:35:20,021::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:35:20,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150888 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:35:27,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:35:27,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000983914 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:35:30,094::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:35:30,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841865 s, 4.9 MB/s\n'; <rc> = 0
>Thread-14738::WARNING::2013-03-17 23:35:30,512::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:35:30,518::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 23:35:37,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:35:37,843::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843239 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:35:40,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:35:40,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141305 s, 290 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:35:40,521::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:35:40,522::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:35:40,945::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:35:40,952::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:35:40,953::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:35:47,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:35:47,880::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859205 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:35:50,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:35:50,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014797 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:35:57,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:35:57,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845211 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:36:00,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:36:00,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872011 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:36:07,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:36:07,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140467 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:36:10,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:36:10,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147277 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:36:17,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:36:18,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756451 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:36:20,742::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:36:20,766::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776126 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:36:28,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:36:28,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159376 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:36:30,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:36:30,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785712 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:36:38,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:36:38,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107639 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:36:40,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:36:40,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00597263 s, 686 kB/s\n'; <rc> = 0
>Thread-14739::WARNING::2013-03-17 23:36:40,986::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:36:40,999::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-17 23:36:48,108::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:36:48,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00072776 s, 5.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:36:51,003::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:36:51,007::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:36:51,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:36:51,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124193 s, 330 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:36:51,439::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:36:51,444::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:36:51,446::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:36:58,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:36:58,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810944 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:37:01,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:37:01,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000986682 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:37:08,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:37:08,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134726 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:37:11,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:37:11,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146598 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:37:18,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:37:18,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000943211 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:37:21,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:37:21,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852977 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:37:28,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:37:28,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135186 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:37:31,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:37:31,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788092 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:37:38,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:37:38,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123621 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:37:41,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:37:41,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166822 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:37:48,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:37:48,382::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135836 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14740::WARNING::2013-03-17 23:37:51,475::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:37:51,479::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:37:51,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:37:51,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000727513 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:37:58,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:37:58,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887288 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:38:01,481::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:38:01,483::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:38:01,592::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:38:01,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153915 s, 266 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:38:02,068::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:38:02,075::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:38:02,076::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:38:08,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:38:08,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885822 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:38:11,712::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 23:38:11,717::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 23:38:11,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:38:11,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00139911 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:38:11,764::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00139911 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 23:38:11,765::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 23:38:11,765::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 23:38:11,766::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 23:38:11,767::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:38:11,768::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:38:11,769::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:38:11,769::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:38:11,769::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:38:11,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:38:12,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:38:12,203::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:38:12,204::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 23:38:12,207::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 23:38:12,208::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 23:38:12,208::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 23:38:12,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:38:12,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:38:12,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:38:12,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174253 s, 235 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:38:18,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:38:18,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000684401 s, 6.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:38:22,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:38:22,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152039 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:38:28,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:38:28,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155406 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:38:32,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:38:32,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103145 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:38:38,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:38:38,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000739197 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:38:42,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:38:42,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0157804 s, 260 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:38:48,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:38:48,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144468 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:38:52,944::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:38:52,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00240475 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:38:58,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:38:58,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134491 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14741::WARNING::2013-03-17 23:39:02,114::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:39:02,118::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:39:03,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:39:03,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000734266 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:39:08,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:39:08,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013672 s, 300 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:39:12,119::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 23:39:12,121::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:39:12,122::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:39:12,528::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:39:12,534::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:39:12,536::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:39:13,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:39:13,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.012175 s, 336 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:39:18,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:39:18,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851451 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:39:23,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:39:23,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791901 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:39:28,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:39:28,801::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146559 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:39:33,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:39:33,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148583 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:39:38,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:39:38,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860514 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:39:43,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:39:43,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000942828 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:39:48,846::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 23:39:48,848::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 23:39:48,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:39:48,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000867924 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:39:48,880::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000867924 s, 2.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 23:39:48,880::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 23:39:48,881::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 23:39:48,882::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 23:39:48,883::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:39:48,884::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:39:48,884::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:39:48,885::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:39:48,885::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:39:48,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:39:49,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:39:49,327::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:39:49,328::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 23:39:49,330::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 23:39:49,331::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 23:39:49,332::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 23:39:49,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:39:49,777::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:39:49,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:39:49,819::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118242 s, 346 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:39:53,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:39:53,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0207874 s, 197 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:39:59,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:39:59,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014813 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:40:03,481::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:40:03,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165692 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:40:09,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:40:09,912::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140313 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14742::WARNING::2013-03-17 23:40:12,562::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:40:12,567::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:40:13,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:40:13,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00202066 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:40:19,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:40:19,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884833 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:40:22,570::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:40:22,571::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:40:22,984::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:40:22,991::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:40:22,992::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:40:23,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:40:23,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125226 s, 327 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:40:29,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:40:29,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801982 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:40:33,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:40:33,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121487 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:40:39,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:40:40,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144629 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:40:43,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:40:43,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873039 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:40:50,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:40:50,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177699 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:40:53,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:40:53,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799342 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:41:00,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:41:00,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155974 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:41:03,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:41:03,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793328 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:41:10,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:41:10,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014524 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:41:14,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:41:14,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188493 s, 217 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:41:20,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:41:20,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868786 s, 4.7 MB/s\n'; <rc> = 0
>Thread-14743::WARNING::2013-03-17 23:41:23,020::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:41:23,022::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:41:24,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:41:24,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138097 s, 297 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:41:30,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:41:30,238::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807801 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:41:33,025::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:41:33,026::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:41:33,449::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:41:33,456::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:41:33,457::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:41:34,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:41:34,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134747 s, 304 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:41:40,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:41:40,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126537 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:41:44,292::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:41:44,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800966 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:41:50,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:41:50,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173078 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:41:54,365::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:41:54,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018842 s, 217 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:42:00,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:42:00,392::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872773 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:42:04,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:42:04,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136828 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:42:10,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:42:10,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877489 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:42:14,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:42:14,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173123 s, 237 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:42:20,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:42:20,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001395 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:42:24,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:42:24,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148921 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:42:30,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:42:30,513::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000969718 s, 4.2 MB/s\n'; <rc> = 0
>Thread-14744::WARNING::2013-03-17 23:42:33,485::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:42:33,489::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:42:34,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:42:34,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016579 s, 247 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:42:40,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:42:40,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837742 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:42:43,491::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:42:43,492::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:42:43,905::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:42:43,912::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:42:43,913::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:42:44,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:42:44,853::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168484 s, 243 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:42:50,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:42:50,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102034 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:42:54,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:42:54,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193322 s, 212 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:43:00,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:43:00,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820525 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:43:05,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:43:05,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00235631 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:43:10,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:43:10,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175573 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:43:15,109::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 23:43:15,112::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 23:43:15,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:43:15,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0012702 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:43:15,162::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0012702 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 23:43:15,163::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 23:43:15,163::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 23:43:15,164::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 23:43:15,166::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:43:15,166::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:43:15,167::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:43:15,167::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:43:15,168::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:43:15,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:43:15,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:43:15,602::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:43:15,603::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 23:43:15,605::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 23:43:15,606::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 23:43:15,607::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 23:43:15,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:43:15,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:43:15,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:43:16,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00940025 s, 436 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:43:20,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:43:20,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0211408 s, 194 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:43:26,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:43:26,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141205 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:43:30,741::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:43:30,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000994096 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:43:36,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:43:36,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120425 s, 340 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:43:40,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:43:40,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781395 s, 5.2 MB/s\n'; <rc> = 0
>Thread-14745::WARNING::2013-03-17 23:43:43,943::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:43:43,947::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:43:46,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:43:46,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834585 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:43:50,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:43:50,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000950436 s, 4.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:43:53,949::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:43:53,950::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:43:54,382::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:43:54,389::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:43:54,390::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:43:56,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:43:56,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00888154 s, 461 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:44:00,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:44:00,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141214 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:44:06,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:44:06,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147557 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:44:10,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:44:10,958::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.012663 s, 323 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:44:16,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:44:16,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152113 s, 269 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:44:20,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:44:20,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829979 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:44:26,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:44:26,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185363 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:44:31,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:44:31,046::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137006 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:44:36,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:44:36,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184252 s, 222 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:44:41,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:44:41,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167404 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:44:46,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:44:46,766::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138653 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:44:51,100::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 23:44:51,101::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 23:44:51,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:44:51,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00107596 s, 1.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:44:51,135::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00107596 s, 1.9 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 23:44:51,136::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 23:44:51,136::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 23:44:51,137::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 23:44:51,138::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:44:51,139::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:44:51,140::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:44:51,140::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:44:51,140::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:44:51,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:44:51,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:44:51,581::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:44:51,582::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 23:44:51,584::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 23:44:51,585::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 23:44:51,586::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 23:44:51,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:44:52,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:44:52,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:44:52,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127235 s, 322 kB/s\n'; <rc> = 0
>Thread-14746::WARNING::2013-03-17 23:44:54,417::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:44:54,420::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:44:56,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:44:56,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132761 s, 309 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:45:02,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:45:02,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0547131 s, 74.9 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:45:04,424::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 23:45:04,426::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:45:04,427::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:45:04,896::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:45:04,902::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:45:04,904::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:45:06,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:45:06,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144679 s, 283 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:45:12,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:45:12,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145107 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:45:16,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:45:17,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799243 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:45:22,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:45:22,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013642 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:45:27,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:45:27,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134611 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:45:32,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:45:32,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867233 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:45:37,129::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:45:37,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788772 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:45:42,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:45:42,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085759 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:45:47,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:45:47,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913612 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:45:52,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:45:52,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825753 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:45:57,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:45:57,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137023 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:46:02,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:46:02,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789689 s, 5.2 MB/s\n'; <rc> = 0
>Thread-14747::WARNING::2013-03-17 23:46:04,931::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:46:04,934::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:46:07,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:46:07,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785787 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:46:12,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:46:12,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00818203 s, 501 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:46:14,937::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:46:14,938::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:46:15,363::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:46:15,370::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:46:15,371::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:46:17,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:46:17,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101191 s, 405 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:46:22,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:46:22,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855128 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:46:27,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:46:27,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840163 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:46:32,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:46:32,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00256889 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:46:37,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:46:37,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016641 s, 246 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:46:42,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:46:42,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840595 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:46:47,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:46:47,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0370271 s, 111 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:46:52,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:46:52,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149687 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:46:57,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:46:57,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138178 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:47:02,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:47:02,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00495836 s, 826 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:47:07,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:47:07,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159496 s, 257 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:47:12,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:47:12,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015456 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14748::WARNING::2013-03-17 23:47:15,398::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:47:15,402::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:47:17,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:47:18,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000711249 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:47:22,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:47:22,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904017 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:47:25,404::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:47:25,406::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:47:25,813::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:47:25,819::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:47:25,821::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:47:28,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:47:28,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.023663 s, 173 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:47:32,717::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:47:32,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152891 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:47:38,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:47:38,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847547 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:47:42,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:47:42,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814244 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:47:48,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:47:48,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756781 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:47:52,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:47:52,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164095 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:47:58,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:47:58,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146145 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:48:02,857::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:48:02,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00370712 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:48:08,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:48:08,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728049 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:48:12,899::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:48:12,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806178 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:48:18,453::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 23:48:18,454::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 23:48:18,455::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:48:18,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00129548 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:48:18,504::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00129548 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 23:48:18,504::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 23:48:18,505::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 23:48:18,507::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 23:48:18,508::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:48:18,509::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:48:18,509::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:48:18,510::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:48:18,510::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:48:18,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:48:18,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:48:18,947::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:48:18,949::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 23:48:18,952::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 23:48:18,953::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 23:48:18,953::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 23:48:18,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:48:19,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:48:19,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:48:19,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139074 s, 295 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:48:22,938::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:48:22,979::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100114 s, 4.1 MB/s\n'; <rc> = 0
>Thread-14749::WARNING::2013-03-17 23:48:25,849::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:48:25,853::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:48:29,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:48:29,450::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825036 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:48:32,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:48:33,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812068 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:48:35,855::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:48:35,856::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:48:36,254::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:48:36,260::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:48:36,262::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:48:39,490::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:48:39,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129745 s, 316 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:48:43,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:48:43,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00870722 s, 470 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:48:49,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:48:49,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892203 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:48:53,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:48:53,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814237 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:48:59,648::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:48:59,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148054 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:49:03,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:49:03,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0044781 s, 915 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:49:09,730::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:49:09,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776364 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:49:13,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:49:13,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115437 s, 355 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:49:19,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:49:19,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147811 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:49:23,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:49:23,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805762 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:49:29,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:49:29,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00290927 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:49:33,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:49:33,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131826 s, 311 kB/s\n'; <rc> = 0
>Thread-14750::WARNING::2013-03-17 23:49:36,289::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:49:36,293::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:49:39,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:49:39,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803248 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:49:43,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:49:43,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102465 s, 4.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:49:46,298::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:49:46,299::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:49:46,725::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:49:46,732::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:49:46,733::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:49:50,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:49:50,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151047 s, 271 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:49:53,466::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 23:49:53,468::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 23:49:53,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:49:53,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000668698 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:49:53,537::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000668698 s, 3.1 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 23:49:53,538::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 23:49:53,538::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 23:49:53,540::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 23:49:53,541::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:49:53,541::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:49:53,542::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:49:53,542::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:49:53,542::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:49:53,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:49:53,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:49:53,978::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:49:53,979::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 23:49:53,982::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 23:49:53,983::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 23:49:53,984::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 23:49:53,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:49:54,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:49:54,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:49:54,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822063 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:50:00,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:50:00,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139315 s, 294 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:50:04,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:50:04,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00748202 s, 547 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:50:10,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:50:10,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00228719 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:50:14,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:50:14,481::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141215 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:50:20,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:50:20,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144018 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:50:24,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:50:24,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000699678 s, 5.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:50:30,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:50:30,409::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796125 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:50:34,524::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:50:34,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148836 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:50:40,455::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:50:40,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138472 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:50:44,571::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:50:44,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826716 s, 5.0 MB/s\n'; <rc> = 0
>Thread-14751::WARNING::2013-03-17 23:50:46,757::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:50:46,761::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:50:50,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:50:50,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153606 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:50:54,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:50:54,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000733178 s, 5.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:50:56,770::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 23:50:56,771::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:50:56,773::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:50:57,213::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:50:57,220::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:50:57,221::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:51:00,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:51:00,671::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010075 s, 407 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:51:04,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:51:04,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000695876 s, 5.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:51:10,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:51:10,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204386 s, 200 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:51:14,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:51:14,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768125 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:51:20,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:51:20,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127248 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:51:24,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:51:24,742::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830129 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:51:30,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:51:30,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128263 s, 319 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:51:34,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:51:34,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134463 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:51:40,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:51:41,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136147 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:51:44,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:51:44,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143533 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:51:51,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:51:51,140::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154709 s, 265 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:51:54,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:51:54,899::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015676 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14752::WARNING::2013-03-17 23:51:57,260::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:51:57,264::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:52:01,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:52:01,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155722 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:52:04,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:52:04,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881448 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:52:07,273::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:52:07,275::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:52:07,761::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:52:07,768::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:52:07,769::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:52:11,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:52:11,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119048 s, 344 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:52:14,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:52:14,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082977 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:52:21,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:52:21,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788235 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:52:24,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:52:25,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810849 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:52:31,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:52:31,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114677 s, 357 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:52:35,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:52:35,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00554523 s, 739 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:52:41,522::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:52:41,548::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000938571 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:52:45,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:52:45,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166525 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:52:51,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:52:51,633::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147057 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:52:55,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:52:55,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145394 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:53:01,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:53:01,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140117 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:53:05,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:53:05,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118041 s, 3.5 MB/s\n'; <rc> = 0
>Thread-14753::WARNING::2013-03-17 23:53:07,794::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:53:07,798::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:53:11,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:53:11,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127888 s, 320 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:53:15,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:53:15,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000926144 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:53:17,800::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:53:17,802::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:53:18,227::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:53:18,234::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:53:18,235::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:53:21,835::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 23:53:21,838::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 23:53:21,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:53:21,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00721634 s, 284 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:53:21,873::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00721634 s, 284 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 23:53:21,873::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 23:53:21,873::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 23:53:21,875::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 23:53:21,876::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:53:21,877::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:53:21,877::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:53:21,878::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:53:21,878::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:53:21,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:53:22,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:53:22,258::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:53:22,259::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 23:53:22,261::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 23:53:22,263::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 23:53:22,263::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 23:53:22,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:53:22,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:53:22,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:53:22,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0068813 s, 595 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:53:25,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:53:25,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0272172 s, 150 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:53:32,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:53:32,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156765 s, 261 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:53:35,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:53:35,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076839 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:53:42,792::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:53:42,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0289327 s, 142 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:53:45,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:53:45,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779555 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:53:52,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:53:52,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131627 s, 311 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:53:55,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:53:55,439::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0389495 s, 105 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:54:03,019::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:54:03,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000949863 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:54:05,445::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:54:05,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811727 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:54:13,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:54:13,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149074 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:54:15,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:54:15,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00562875 s, 728 kB/s\n'; <rc> = 0
>Thread-14754::WARNING::2013-03-17 23:54:18,273::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:54:18,280::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:54:23,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:54:23,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000939059 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:54:25,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:54:25,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144671 s, 2.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:54:28,282::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:54:28,283::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:54:28,725::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:54:28,732::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:54:28,734::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:54:33,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:54:33,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148182 s, 276 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:54:35,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:54:35,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860589 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:54:43,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:54:43,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776689 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:54:45,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:54:45,709::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829179 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:54:53,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:54:53,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167718 s, 244 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:54:55,713::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 23:54:55,715::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 23:54:55,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:54:55,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0173438 s, 118 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:54:55,772::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0173438 s, 118 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 23:54:55,773::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 23:54:55,773::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 23:54:55,775::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 23:54:55,776::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:54:55,777::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:54:55,777::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:54:55,777::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:54:55,778::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:54:55,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:54:56,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:54:56,224::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:54:56,225::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 23:54:56,228::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 23:54:56,229::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 23:54:56,229::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 23:54:56,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:54:56,622::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:54:56,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:54:56,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179743 s, 228 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:55:03,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:55:03,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181241 s, 226 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:55:06,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:55:06,756::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141184 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:55:13,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:55:13,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183694 s, 223 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:55:16,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:55:16,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837363 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:55:23,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:55:23,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152808 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:55:26,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:55:26,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000708765 s, 5.8 MB/s\n'; <rc> = 0
>Thread-14755::WARNING::2013-03-17 23:55:28,768::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:55:28,775::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:55:33,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:55:33,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192638 s, 213 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:55:36,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:55:36,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801028 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:55:38,778::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:55:38,780::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:55:39,179::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:55:39,186::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:55:39,187::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:55:43,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:55:43,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0198905 s, 206 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:55:46,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:55:46,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00328023 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:55:53,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:55:54,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137142 s, 299 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:55:56,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:55:57,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125819 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:56:04,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:56:04,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101952 s, 402 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:56:07,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:56:07,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150078 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:56:14,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:56:14,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165581 s, 247 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:56:17,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:56:17,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112711 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:56:24,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:56:24,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117264 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:56:27,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:56:27,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139198 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:56:34,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:56:34,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145051 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:56:37,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:56:37,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105227 s, 3.9 MB/s\n'; <rc> = 0
>Thread-14756::WARNING::2013-03-17 23:56:39,214::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:56:39,218::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:56:44,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:56:44,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788369 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:56:47,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:56:47,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000951631 s, 4.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:56:49,220::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-17 23:56:49,221::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:56:49,222::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:56:49,662::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:56:49,668::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:56:49,670::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:56:54,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:56:54,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00847373 s, 483 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:56:57,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:56:57,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00586493 s, 698 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:57:04,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:57:04,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876304 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:57:07,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:57:07,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138816 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:57:14,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:57:14,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852076 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:57:17,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:57:17,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180522 s, 227 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:57:24,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:57:24,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000730547 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:57:27,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:57:27,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140322 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:57:34,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:57:34,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875555 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:57:37,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:57:37,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135102 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:57:44,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:57:44,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140428 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:57:47,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:57:47,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898946 s, 4.6 MB/s\n'; <rc> = 0
>Thread-14757::WARNING::2013-03-17 23:57:49,699::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:57:49,703::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:57:54,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:57:54,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117925 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:57:57,537::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:57:57,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00799665 s, 512 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:57:59,705::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:57:59,707::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:58:00,142::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:58:00,149::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:58:00,150::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:58:04,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:58:05,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192041 s, 213 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:58:07,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:58:07,611::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00264293 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:58:15,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:58:15,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000960641 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:58:17,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:58:17,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00855601 s, 479 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:58:25,144::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-17 23:58:25,145::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-17 23:58:25,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:58:25,175::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000892872 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:58:25,181::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000892872 s, 2.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-17 23:58:25,181::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-17 23:58:25,182::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-17 23:58:25,183::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-17 23:58:25,185::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:58:25,185::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:58:25,186::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:58:25,186::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:58:25,187::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-17 23:58:25,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:58:25,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:58:25,605::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:58:25,606::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-17 23:58:25,608::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-17 23:58:25,610::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-17 23:58:25,610::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-17 23:58:25,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:58:25,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:58:25,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:58:26,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00670237 s, 611 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:58:27,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:58:27,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00308049 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:58:36,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:58:36,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145545 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:58:37,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:58:37,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125197 s, 327 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:58:46,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:58:46,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0087287 s, 469 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:58:47,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:58:47,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138325 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:58:56,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:58:56,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113518 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:58:57,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:58:57,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104552 s, 3.9 MB/s\n'; <rc> = 0
>Thread-14758::WARNING::2013-03-17 23:59:00,189::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-17 23:59:00,194::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-17 23:59:06,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:59:06,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879351 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:59:07,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:59:07,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00099456 s, 4.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-17 23:59:10,201::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-17 23:59:10,202::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-17 23:59:10,623::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-17 23:59:10,629::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-17 23:59:10,631::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-17 23:59:16,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:59:16,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117225 s, 349 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:59:17,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:59:17,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175094 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:59:26,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:59:26,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141669 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:59:27,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:59:27,979::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139657 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:59:36,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:59:36,569::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089444 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:59:37,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:59:38,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154292 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:59:46,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:59:46,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00257193 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:59:48,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:59:48,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109704 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-17 23:59:56,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-17 23:59:56,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776048 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:59:58,117::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-17 23:59:58,118::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-17 23:59:58,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:59:58,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00592263 s, 346 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:59:58,157::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00592263 s, 346 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-17 23:59:58,157::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-17 23:59:58,157::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-17 23:59:58,159::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-17 23:59:58,160::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:59:58,160::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:59:58,160::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:59:58,161::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:59:58,161::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-17 23:59:58,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:59:58,591::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:59:58,595::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-17 23:59:58,596::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-17 23:59:58,598::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-17 23:59:58,600::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-17 23:59:58,600::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-17 23:59:58,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:59:59,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-17 23:59:59,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-17 23:59:59,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139866 s, 293 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:00:06,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:00:06,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0354906 s, 115 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:00:09,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:00:09,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00297746 s, 1.4 MB/s\n'; <rc> = 0
>Thread-14759::WARNING::2013-03-18 00:00:10,656::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:00:10,660::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 00:00:16,875::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:00:16,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843897 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:00:19,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:00:19,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000643363 s, 6.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:00:20,662::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:00:20,664::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:00:21,172::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:00:21,179::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:00:21,181::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:00:26,944::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:00:26,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100336 s, 408 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:00:29,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:00:29,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773254 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:00:37,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:00:37,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847168 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:00:39,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:00:39,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00190095 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:00:47,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:00:47,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836064 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:00:49,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:00:49,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154932 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:00:57,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:00:57,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143743 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:00:59,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:00:59,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144139 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:01:07,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:01:07,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146284 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:01:09,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:01:09,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152607 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:01:17,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:01:17,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00341736 s, 1.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:01:19,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:01:19,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118445 s, 3.5 MB/s\n'; <rc> = 0
>Thread-14760::WARNING::2013-03-18 00:01:21,210::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:01:21,216::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 00:01:27,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:01:27,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825377 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:01:29,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:01:29,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776232 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:01:31,219::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:01:31,220::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:01:31,649::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:01:31,656::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:01:31,657::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:01:37,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:01:37,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0197339 s, 208 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:01:39,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:01:39,512::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799684 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:01:47,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:01:47,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008037 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:01:49,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:01:49,548::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800329 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:01:57,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:01:57,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140396 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:01:59,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:01:59,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001452 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:02:07,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:02:07,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078396 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:02:09,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:02:09,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135767 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:02:17,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:02:17,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140893 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:02:19,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:02:19,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898339 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:02:27,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:02:27,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160368 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:02:29,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:02:29,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867184 s, 4.7 MB/s\n'; <rc> = 0
>Thread-14761::WARNING::2013-03-18 00:02:31,684::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:02:31,688::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 00:02:37,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:02:38,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832327 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:02:39,742::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:02:39,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00093358 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:02:41,692::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 00:02:41,693::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:02:41,694::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:02:42,138::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:02:42,144::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:02:42,146::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:02:48,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:02:48,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166298 s, 246 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:02:49,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:02:49,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141957 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:02:58,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:02:58,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166609 s, 246 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:02:59,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:02:59,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854461 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:03:08,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:03:08,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814191 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:03:09,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:03:09,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142761 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:03:18,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:03:18,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00190196 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:03:19,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:03:19,928::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882856 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:03:28,377::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 00:03:28,378::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 00:03:28,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:03:28,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000819114 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:03:28,408::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000819114 s, 2.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 00:03:28,409::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 00:03:28,409::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 00:03:28,411::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 00:03:28,412::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:03:28,412::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:03:28,414::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:03:28,414::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:03:28,414::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:03:28,415::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:03:28,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:03:28,862::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:03:28,863::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 00:03:28,866::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 00:03:28,867::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 00:03:28,867::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 00:03:28,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:03:29,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:03:29,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:03:29,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00804224 s, 509 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:03:29,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:03:29,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140135 s, 292 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:03:39,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:03:39,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860202 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:03:39,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:03:40,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000721783 s, 5.7 MB/s\n'; <rc> = 0
>Thread-14762::WARNING::2013-03-18 00:03:42,170::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:03:42,173::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 00:03:49,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:03:49,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126096 s, 325 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:03:50,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:03:50,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826892 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:03:52,175::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:03:52,177::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:03:52,635::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:03:52,642::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:03:52,643::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:03:59,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:03:59,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175982 s, 233 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:04:00,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:04:00,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017765 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:04:09,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:04:09,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915337 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:04:10,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:04:10,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869786 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:04:19,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:04:19,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816622 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:04:20,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:04:20,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000933705 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:04:29,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:04:29,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173279 s, 236 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:04:30,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:04:30,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145811 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:04:39,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:04:39,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134428 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:04:40,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:04:40,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857053 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:04:49,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:04:49,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192087 s, 213 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:04:50,281::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:04:50,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158113 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14763::WARNING::2013-03-18 00:04:52,669::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:04:52,673::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 00:04:59,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:05:00,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749499 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:05:00,327::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 00:05:00,329::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 00:05:00,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:05:00,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000776051 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:05:00,355::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000776051 s, 2.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 00:05:00,356::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 00:05:00,356::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 00:05:00,358::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 00:05:00,359::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:05:00,359::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:05:00,360::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:05:00,360::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:05:00,361::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:05:00,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:05:00,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:05:00,787::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:05:00,788::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 00:05:00,791::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 00:05:00,792::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 00:05:00,792::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 00:05:00,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:05:01,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:05:01,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:05:01,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0093632 s, 437 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:05:02,681::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:05:02,682::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:05:03,124::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:05:03,130::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:05:03,132::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:05:10,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:05:10,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184218 s, 222 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:05:11,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:05:11,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837793 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:05:20,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:05:20,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784958 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:05:21,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:05:21,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134014 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:05:30,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:05:30,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130511 s, 314 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:05:31,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:05:31,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00286132 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:05:40,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:05:40,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154708 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:05:41,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:05:41,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151534 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:05:50,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:05:50,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126355 s, 324 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:05:51,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:05:51,402::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133609 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:06:00,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:06:00,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818362 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:06:01,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:06:01,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151286 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14764::WARNING::2013-03-18 00:06:03,164::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:06:03,170::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 00:06:10,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:06:10,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00876849 s, 467 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:06:11,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:06:11,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157941 s, 2.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:06:13,174::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:06:13,176::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:06:13,629::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:06:13,636::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:06:13,637::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:06:20,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:06:20,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00638155 s, 642 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:06:21,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:06:21,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176048 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:06:30,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:06:30,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156203 s, 262 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:06:31,614::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:06:31,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117328 s, 349 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:06:40,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:06:40,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00072674 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:06:41,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:06:41,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890656 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:06:50,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:06:50,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133921 s, 306 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:06:51,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:06:51,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127335 s, 322 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:07:00,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:07:00,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147294 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:07:01,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:07:01,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00295398 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:07:11,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:07:11,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115015 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:07:11,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:07:11,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161124 s, 254 kB/s\n'; <rc> = 0
>Thread-14765::WARNING::2013-03-18 00:07:13,663::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:07:13,667::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 00:07:21,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:07:21,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835774 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:07:21,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:07:21,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100094 s, 4.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:07:23,669::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:07:23,670::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:07:24,115::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:07:24,122::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:07:24,123::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:07:31,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:07:31,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778791 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:07:31,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:07:31,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121894 s, 336 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:07:41,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:07:41,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847604 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:07:41,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:07:41,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796928 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:07:51,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:07:51,347::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837629 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:07:51,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:07:52,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811296 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:08:01,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:08:01,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142332 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:08:02,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:08:02,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768818 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:08:11,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:08:11,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131518 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:08:12,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:08:12,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134358 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:08:21,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:08:21,622::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907263 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:08:22,128::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:08:22,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105733 s, 3.9 MB/s\n'; <rc> = 0
>Thread-14766::WARNING::2013-03-18 00:08:24,149::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:08:24,152::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 00:08:31,667::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 00:08:31,668::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 00:08:31,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:08:31,709::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00141837 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:08:31,710::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00141837 s, 1.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 00:08:31,711::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 00:08:31,711::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 00:08:31,713::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 00:08:31,714::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:08:31,715::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:08:31,715::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:08:31,715::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:08:31,716::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:08:31,717::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:08:32,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:08:32,147::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:08:32,148::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 00:08:32,151::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 00:08:32,152::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 00:08:32,153::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 00:08:32,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:08:32,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:08:32,270::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0456798 s, 89.7 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:08:32,658::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:08:32,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:08:32,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00539895 s, 759 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:08:34,154::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 00:08:34,155::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:08:34,156::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:08:34,605::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:08:34,615::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:08:34,617::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:08:42,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:08:42,311::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746664 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:08:42,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:08:42,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156058 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:08:52,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:08:52,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789649 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:08:52,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:08:52,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100972 s, 406 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:09:02,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:09:02,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128783 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:09:02,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:09:02,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141928 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:09:12,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:09:12,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124398 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:09:12,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:09:13,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134405 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:09:22,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:09:22,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155114 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:09:23,070::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:09:23,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124464 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:09:32,492::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:09:32,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000941732 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:09:33,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:09:33,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00869445 s, 471 kB/s\n'; <rc> = 0
>Thread-14767::WARNING::2013-03-18 00:09:34,648::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:09:34,654::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:09:42,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:09:42,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923442 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:09:43,281::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:09:43,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777303 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:09:44,656::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:09:44,658::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:09:45,118::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:09:45,125::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:09:45,126::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:09:52,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:09:52,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845965 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:09:53,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:09:53,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820312 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:10:02,604::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 00:10:02,605::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 00:10:02,606::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:10:02,657::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00226085 s, 906 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:10:02,658::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00226085 s, 906 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 00:10:02,658::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 00:10:02,659::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 00:10:02,660::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 00:10:02,661::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:10:02,662::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:10:02,662::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:10:02,663::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:10:02,663::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:10:02,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:10:03,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:10:03,096::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:10:03,097::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 00:10:03,100::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 00:10:03,101::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 00:10:03,101::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 00:10:03,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:10:03,412::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:10:03,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:10:03,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:10:03,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130869 s, 313 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:10:03,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159118 s, 257 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:10:13,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:10:13,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:10:13,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148491 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:10:13,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000991695 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:10:23,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:10:23,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:10:23,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846343 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:10:23,791::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744047 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:10:33,797::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:10:33,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:10:33,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102874 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:10:33,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106343 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:10:43,933::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:10:44,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:10:44,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0301505 s, 136 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:10:44,291::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00072097 s, 5.7 MB/s\n'; <rc> = 0
>Thread-14768::WARNING::2013-03-18 00:10:45,164::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:10:45,170::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:10:54,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:10:54,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843247 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:10:54,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:10:54,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901849 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:10:55,174::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:10:55,176::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:10:55,622::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:10:55,629::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:10:55,630::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:11:04,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:11:04,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777305 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:11:04,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:11:04,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106615 s, 384 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:11:14,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:11:14,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011868 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:11:14,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:11:14,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814686 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:11:24,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:11:24,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136154 s, 301 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:11:24,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:11:24,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00734262 s, 558 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:11:34,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:11:34,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771313 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:11:35,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:11:35,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151025 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:11:44,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:11:44,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842345 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:11:45,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:11:45,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0217725 s, 188 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:11:54,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:11:54,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860407 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:11:55,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:11:55,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013677 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14769::WARNING::2013-03-18 00:11:55,659::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:11:55,663::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:12:04,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:12:04,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840272 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:12:05,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:12:05,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0068814 s, 595 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:12:05,666::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:12:05,668::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:12:06,069::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:12:06,076::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:12:06,077::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:12:14,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:12:14,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016863 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:12:15,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:12:15,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138238 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:12:24,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:12:24,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844225 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:12:25,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:12:25,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850065 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:12:34,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:12:35,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142872 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:12:35,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:12:35,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740844 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:12:45,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:12:45,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144448 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:12:45,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:12:45,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148394 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:12:55,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:12:55,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169905 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:12:55,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:12:55,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149183 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:13:05,117::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:13:05,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00509081 s, 805 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:13:05,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:13:05,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144356 s, 284 kB/s\n'; <rc> = 0
>Thread-14770::WARNING::2013-03-18 00:13:06,116::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:13:06,119::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:13:15,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:13:15,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864193 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:13:16,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:13:16,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00959214 s, 427 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:13:16,121::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:13:16,123::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:13:16,639::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:13:16,645::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:13:16,646::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:13:25,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:13:25,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161309 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:13:26,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:13:26,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129575 s, 316 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:13:35,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:13:35,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013434 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:13:36,267::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 00:13:36,269::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 00:13:36,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:13:36,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00079094 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:13:36,300::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00079094 s, 2.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 00:13:36,300::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 00:13:36,301::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 00:13:36,303::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 00:13:36,304::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:13:36,305::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:13:36,306::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:13:36,306::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:13:36,307::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:13:36,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:13:36,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:13:36,749::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:13:36,750::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 00:13:36,754::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 00:13:36,755::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 00:13:36,756::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 00:13:36,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:13:37,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:13:37,129::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:13:37,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124298 s, 330 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:13:45,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:13:45,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150532 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:13:47,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:13:47,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00919311 s, 446 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:13:55,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:13:55,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817683 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:13:57,292::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:13:57,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176794 s, 232 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:14:05,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:14:05,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771739 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:14:07,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:14:07,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00933549 s, 439 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:14:15,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:14:15,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01852 s, 221 kB/s\n'; <rc> = 0
>Thread-14771::WARNING::2013-03-18 00:14:16,671::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:14:16,675::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 00:14:17,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:14:17,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792279 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:14:25,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:14:25,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000951466 s, 4.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:14:26,677::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 00:14:26,678::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:14:26,680::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:14:27,157::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:14:27,165::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:14:27,166::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:14:27,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:14:27,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010617 s, 386 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:14:35,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:14:35,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000706945 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:14:37,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:14:37,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789244 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:14:45,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:14:45,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915675 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:14:47,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:14:47,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181899 s, 225 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:14:55,760::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:14:55,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832318 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:14:57,814::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:14:57,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743457 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:15:05,793::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 00:15:05,794::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 00:15:05,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:15:05,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00130831 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:15:05,832::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00130831 s, 1.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 00:15:05,832::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 00:15:05,833::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 00:15:05,834::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 00:15:05,835::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:15:05,836::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:15:05,836::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:15:05,837::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:15:05,837::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:15:05,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:15:06,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:15:06,292::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:15:06,293::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 00:15:06,295::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 00:15:06,296::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 00:15:06,297::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 00:15:06,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:15:06,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:15:06,648::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:15:06,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00611208 s, 670 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:15:07,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:15:07,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00929103 s, 441 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:15:16,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:15:16,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019336 s, 212 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:15:17,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:15:18,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143077 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:15:26,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:15:26,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000739168 s, 5.5 MB/s\n'; <rc> = 0
>Thread-14772::WARNING::2013-03-18 00:15:27,189::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:15:27,193::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 00:15:28,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:15:28,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0239888 s, 171 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:15:36,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:15:36,880::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117954 s, 347 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:15:37,196::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:15:37,198::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:15:37,609::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:15:37,615::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:15:37,617::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:15:38,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:15:38,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194833 s, 210 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:15:46,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:15:46,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876158 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:15:48,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:15:48,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140057 s, 292 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:15:56,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:15:56,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.002159 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:15:58,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:15:58,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150398 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:16:06,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:16:07,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00191776 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:16:08,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:16:08,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186843 s, 219 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:16:17,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:16:17,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160469 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:16:18,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:16:18,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137931 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:16:27,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:16:27,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145375 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:16:28,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:16:28,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135999 s, 301 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:16:37,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:16:37,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112992 s, 3.6 MB/s\n'; <rc> = 0
>Thread-14773::WARNING::2013-03-18 00:16:37,645::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:16:37,648::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 00:16:38,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:16:38,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806895 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:16:47,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:16:47,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737102 s, 5.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:16:47,651::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:16:47,652::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:16:48,045::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:16:48,052::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:16:48,053::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:16:48,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:16:48,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139332 s, 294 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:16:57,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:16:57,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014343 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:16:58,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:16:58,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793571 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:17:07,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:17:07,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127197 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:17:08,899::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:17:08,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136926 s, 299 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:17:17,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:17:17,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808331 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:17:18,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:17:19,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805851 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:17:27,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:17:27,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157373 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:17:29,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:17:29,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163658 s, 250 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:17:37,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:17:37,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153557 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:17:39,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:17:39,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150106 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:17:47,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:17:47,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00704103 s, 582 kB/s\n'; <rc> = 0
>Thread-14774::WARNING::2013-03-18 00:17:48,081::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:17:48,092::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 00:17:49,227::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:17:49,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0201731 s, 203 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:17:57,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:17:57,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00071179 s, 5.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:17:58,096::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:17:58,097::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:17:58,580::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:17:58,586::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:17:58,588::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:17:59,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:17:59,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.012822 s, 319 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:18:07,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:18:07,765::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014216 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:18:09,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:18:09,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121861 s, 336 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:18:17,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:18:17,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808364 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:18:19,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:18:19,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074886 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:18:27,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:18:27,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887471 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:18:29,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:18:29,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152934 s, 268 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:18:37,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:18:37,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150914 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:18:39,655::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 00:18:39,657::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 00:18:39,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:18:39,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0012912 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:18:39,706::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0012912 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 00:18:39,707::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 00:18:39,707::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 00:18:39,708::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 00:18:39,709::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:18:39,710::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:18:39,711::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:18:39,711::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:18:39,711::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:18:39,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:18:40,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:18:40,159::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:18:40,160::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 00:18:40,163::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 00:18:40,164::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 00:18:40,164::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 00:18:40,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:18:40,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:18:40,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:18:40,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104546 s, 392 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:18:47,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:18:47,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139735 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:18:50,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:18:50,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013431 s, 305 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:18:58,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:18:58,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000661711 s, 6.2 MB/s\n'; <rc> = 0
>Thread-14775::WARNING::2013-03-18 00:18:58,615::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:18:58,619::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 00:19:00,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:19:00,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191549 s, 214 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:19:08,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:19:08,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915293 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:19:08,630::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:19:08,631::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:19:09,049::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:19:09,055::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:19:09,057::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:19:10,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:19:10,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124334 s, 329 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:19:18,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:19:18,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016515 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:19:20,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:19:20,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808423 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:19:28,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:19:28,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777926 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:19:30,944::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:19:30,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138313 s, 296 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:19:38,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:19:38,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013438 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:19:41,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:19:41,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128953 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:19:48,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:19:48,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149914 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:19:51,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:19:51,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140834 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:19:58,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:19:58,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808626 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:20:01,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:20:01,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867118 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:20:08,479::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 00:20:08,480::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 00:20:08,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:20:08,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000918724 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:20:08,555::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000918724 s, 2.2 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 00:20:08,558::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 00:20:08,559::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 00:20:08,563::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 00:20:08,568::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:20:08,571::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:20:08,572::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:20:08,574::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:20:08,576::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:20:08,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:20:09,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:20:09,077::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:20:09,078::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 00:20:09,081::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 00:20:09,083::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 00:20:09,085::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-14776::WARNING::2013-03-18 00:20:09,091::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:20:09,102::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:20:09,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:20:09,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:20:09,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:20:09,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00440529 s, 930 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:20:11,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:20:11,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190566 s, 215 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:20:19,109::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 00:20:19,110::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:20:19,111::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:20:19,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:20:19,537::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:20:19,544::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:20:19,546::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:20:19,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0098192 s, 417 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:20:21,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:20:21,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128792 s, 318 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:20:29,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:20:29,622::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800332 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:20:31,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:20:31,478::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000507816 s, 8.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:20:39,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:20:39,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135194 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:20:41,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:20:41,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0007496 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:20:49,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:20:49,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759784 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:20:51,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:20:51,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799054 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:20:59,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:20:59,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126782 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:21:01,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:21:01,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786083 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:21:09,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:21:09,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877508 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:21:11,732::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:21:11,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123236 s, 3.3 MB/s\n'; <rc> = 0
>Thread-14777::WARNING::2013-03-18 00:21:19,598::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:21:19,602::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:21:19,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:21:19,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174112 s, 235 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:21:21,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:21:21,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082045 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:21:29,604::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:21:29,605::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:21:29,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:21:29,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159476 s, 257 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:21:30,086::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:21:30,091::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:21:30,093::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:21:31,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:21:31,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123695 s, 331 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:21:39,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:21:39,979::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151329 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:21:41,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:21:42,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087442 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:21:49,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:21:50,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000905853 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:21:52,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:21:52,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00232206 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:22:00,021::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:22:00,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014565 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:22:02,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:22:02,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139383 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:22:10,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:22:10,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834217 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:22:12,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:22:12,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000726085 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:22:20,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:22:20,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133555 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:22:22,281::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:22:22,333::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00322058 s, 1.3 MB/s\n'; <rc> = 0
>Thread-14778::WARNING::2013-03-18 00:22:30,132::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:22:30,136::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:22:30,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:22:30,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791512 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:22:32,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:22:32,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815302 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:22:40,139::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:22:40,140::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:22:40,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:22:40,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158488 s, 258 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:22:40,634::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:22:40,639::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:22:40,641::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:22:42,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:22:42,490::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109035 s, 376 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:22:50,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:22:50,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857619 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:22:52,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:22:52,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854124 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:23:00,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:23:00,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758753 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:23:02,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:23:02,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019162 s, 214 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:23:10,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:23:10,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806204 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:23:12,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:23:12,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137197 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:23:20,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:23:20,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000927022 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:23:22,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:23:22,820::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119539 s, 343 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:23:30,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:23:30,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015212 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:23:32,863::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:23:32,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133109 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:23:40,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:23:40,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808105 s, 5.1 MB/s\n'; <rc> = 0
>Thread-14779::WARNING::2013-03-18 00:23:40,671::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:23:40,683::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 00:23:42,948::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 00:23:42,950::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 00:23:42,950::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:23:42,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00992302 s, 206 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:23:42,990::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00992302 s, 206 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 00:23:42,991::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 00:23:42,991::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 00:23:42,993::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 00:23:42,994::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:23:42,995::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:23:42,995::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:23:42,996::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:23:42,996::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:23:42,997::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:23:43,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:23:43,438::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:23:43,439::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 00:23:43,441::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 00:23:43,442::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 00:23:43,443::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 00:23:43,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:23:43,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:23:43,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:23:43,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083593 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:23:50,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:23:50,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126817 s, 323 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:23:50,686::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:23:50,688::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:23:51,122::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:23:51,129::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:23:51,130::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:23:53,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:23:53,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000922293 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:24:00,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:24:00,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752563 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:24:03,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:24:04,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107419 s, 381 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:24:10,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:24:10,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152056 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:24:14,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:24:14,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0274062 s, 149 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:24:20,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:24:20,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118815 s, 345 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:24:24,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:24:24,238::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137623 s, 298 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:24:30,760::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:24:30,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084319 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:24:34,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:24:34,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00465758 s, 879 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:24:40,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:24:40,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000718592 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:24:44,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:24:44,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126236 s, 324 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:24:50,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:24:50,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829026 s, 4.9 MB/s\n'; <rc> = 0
>Thread-14780::WARNING::2013-03-18 00:24:51,160::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:24:51,166::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 00:24:54,492::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:24:54,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900207 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:25:00,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:25:00,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00926269 s, 442 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:25:01,170::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:25:01,172::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:25:01,636::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:25:01,642::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:25:01,644::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:25:04,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:25:04,598::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133783 s, 306 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:25:10,944::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 00:25:10,945::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 00:25:10,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:25:10,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000953953 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:25:10,977::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000953953 s, 2.1 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 00:25:10,978::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 00:25:10,978::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 00:25:10,980::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 00:25:10,981::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:25:10,981::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:25:10,982::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:25:10,982::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:25:10,982::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:25:10,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:25:11,395::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:25:11,399::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:25:11,400::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 00:25:11,402::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 00:25:11,404::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 00:25:11,404::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 00:25:11,405::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:25:11,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:25:11,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:25:11,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0318181 s, 129 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:25:14,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:25:14,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00931116 s, 440 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:25:21,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:25:21,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782234 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:25:24,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:25:24,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0243224 s, 168 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:25:31,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:25:31,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906721 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:25:34,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:25:34,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012309 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:25:41,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:25:41,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145743 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:25:44,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:25:44,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140865 s, 291 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:25:51,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:25:52,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014012 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:25:54,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:25:55,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185552 s, 2.2 MB/s\n'; <rc> = 0
>Thread-14781::WARNING::2013-03-18 00:26:01,669::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:26:01,673::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:26:02,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:26:02,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155771 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:26:05,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:26:05,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0219001 s, 187 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:26:11,675::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 00:26:11,676::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:26:11,677::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:26:12,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:26:12,123::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:26:12,132::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:26:12,134::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:26:12,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173891 s, 236 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:26:15,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:26:15,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153401 s, 267 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:26:22,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:26:22,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877138 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:26:25,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:26:25,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178884 s, 229 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:26:32,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:26:32,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832633 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:26:35,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:26:35,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106834 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:26:42,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:26:42,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119607 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:26:45,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:26:45,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145131 s, 282 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:26:52,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:26:52,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015945 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:26:55,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:26:55,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822043 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:27:02,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:27:02,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145052 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:27:05,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:27:05,611::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182882 s, 224 kB/s\n'; <rc> = 0
>Thread-14782::WARNING::2013-03-18 00:27:12,165::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:27:12,170::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:27:12,373::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:27:12,402::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805323 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:27:15,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:27:15,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794467 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:27:22,172::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:27:22,173::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:27:22,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:27:22,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142073 s, 288 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:27:22,717::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:27:22,722::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:27:22,723::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:27:25,717::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:27:25,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0200286 s, 205 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:27:32,482::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:27:32,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000918916 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:27:35,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:27:35,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000931003 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:27:42,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:27:42,548::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885135 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:27:45,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:27:45,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184146 s, 222 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:27:52,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:27:52,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855866 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:27:55,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:27:55,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850981 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:28:02,596::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:28:02,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000965581 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:28:06,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:28:06,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0168145 s, 244 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:28:12,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:28:12,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141577 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:28:16,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:28:16,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00181566 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:28:22,681::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:28:22,733::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153149 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14783::WARNING::2013-03-18 00:28:22,747::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:28:22,751::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 00:28:26,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:28:26,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01779 s, 230 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:28:32,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:28:32,772::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:28:32,775::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:28:32,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015006 s, 273 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:28:33,260::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:28:33,265::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:28:33,266::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:28:36,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:28:36,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172253 s, 238 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:28:42,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:28:42,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102408 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:28:46,398::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 00:28:46,399::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 00:28:46,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:28:46,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.010903 s, 188 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:28:46,438::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.010903 s, 188 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 00:28:46,439::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 00:28:46,439::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 00:28:46,441::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 00:28:46,443::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:28:46,444::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:28:46,444::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:28:46,445::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:28:46,445::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:28:46,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:28:46,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:28:46,848::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:28:46,849::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 00:28:46,852::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 00:28:46,853::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 00:28:46,853::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 00:28:46,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:28:47,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:28:47,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:28:47,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00904834 s, 453 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:28:52,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:28:52,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152295 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:28:57,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:28:57,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826614 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:29:02,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:29:02,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880244 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:29:07,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:29:07,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813011 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:29:12,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:29:12,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013883 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:29:17,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:29:17,700::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0215841 s, 190 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:29:22,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:29:23,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00412705 s, 992 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:29:27,741::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:29:27,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014769 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:29:33,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:29:33,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737387 s, 5.6 MB/s\n'; <rc> = 0
>Thread-14784::WARNING::2013-03-18 00:29:33,293::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:29:33,301::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 00:29:37,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:29:37,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830642 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:29:43,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:29:43,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000949844 s, 4.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:29:43,306::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:29:43,308::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:29:43,736::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:29:43,743::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:29:43,744::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:29:48,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:29:48,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185637 s, 221 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:29:53,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:29:53,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00692454 s, 592 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:29:58,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:29:58,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00957656 s, 428 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:30:03,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:30:03,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126003 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:30:08,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:30:08,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0262628 s, 156 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:30:13,236::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 00:30:13,237::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 00:30:13,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:30:13,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00137831 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:30:13,275::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00137831 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 00:30:13,276::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 00:30:13,276::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 00:30:13,278::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 00:30:13,279::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:30:13,279::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:30:13,280::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:30:13,280::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:30:13,280::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:30:13,281::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:30:13,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:30:13,735::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:30:13,736::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 00:30:13,738::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 00:30:13,739::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 00:30:13,740::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 00:30:13,741::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:30:14,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:30:14,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:30:14,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112293 s, 365 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:30:18,445::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:30:18,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0094196 s, 435 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:30:24,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:30:24,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00960865 s, 426 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:30:28,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:30:28,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149788 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:30:34,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:30:34,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139455 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:30:38,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:30:38,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118971 s, 3.4 MB/s\n'; <rc> = 0
>Thread-14785::WARNING::2013-03-18 00:30:43,770::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:30:43,776::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:30:44,254::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:30:44,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105888 s, 387 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:30:48,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:30:48,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911994 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:30:53,779::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:30:53,781::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:30:54,194::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:30:54,200::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:30:54,201::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:30:54,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:30:54,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0204359 s, 200 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:30:58,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:30:58,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00099296 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:31:04,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:31:04,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.028297 s, 145 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:31:08,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:31:09,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000727916 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:31:14,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:31:14,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833931 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:31:19,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:31:19,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758402 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:31:24,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:31:24,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013047 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:31:29,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:31:29,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0021796 s, 1.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:31:34,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:31:34,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141774 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:31:39,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:31:39,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00307114 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:31:44,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:31:44,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124173 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:31:49,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:31:49,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133101 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14786::WARNING::2013-03-18 00:31:54,237::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:31:54,241::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:31:54,592::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:31:54,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129937 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:31:59,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:31:59,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0222876 s, 184 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:32:04,243::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 00:32:04,244::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:32:04,245::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:32:04,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:32:04,686::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:32:04,693::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:32:04,694::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:32:04,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0258169 s, 159 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:32:09,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:32:09,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00269205 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:32:14,732::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:32:14,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148395 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:32:19,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:32:19,617::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128942 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:32:24,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:32:24,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000710507 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:32:29,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:32:29,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139805 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:32:34,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:32:34,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116086 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:32:39,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:32:39,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761289 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:32:44,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:32:44,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128682 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:32:49,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:32:49,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141216 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:32:54,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:32:54,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000951402 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:32:59,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:32:59,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00643066 s, 637 kB/s\n'; <rc> = 0
>Thread-14787::WARNING::2013-03-18 00:33:04,734::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:33:04,736::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:33:05,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:33:05,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806197 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:33:09,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:33:10,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888766 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:33:14,749::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:33:14,750::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:33:15,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:33:15,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167222 s, 245 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:33:15,209::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:33:15,214::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:33:15,215::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:33:20,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:33:20,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.032622 s, 126 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:33:25,099::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:33:25,140::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906252 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:33:30,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:33:30,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00581348 s, 705 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:33:35,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:33:35,191::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104729 s, 391 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:33:40,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:33:40,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001087 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:33:45,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:33:45,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000921578 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:33:50,316::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 00:33:50,318::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 00:33:50,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:33:50,364::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00150865 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:33:50,368::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00150865 s, 1.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 00:33:50,368::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 00:33:50,369::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 00:33:50,370::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 00:33:50,371::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:33:50,372::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:33:50,372::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:33:50,372::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:33:50,373::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:33:50,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:33:50,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:33:50,799::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:33:50,800::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 00:33:50,802::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 00:33:50,804::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 00:33:50,804::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 00:33:50,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:33:51,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:33:51,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:33:51,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185509 s, 221 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:33:55,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:33:55,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166029 s, 247 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:34:01,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:34:01,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014777 s, 277 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:34:05,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:34:05,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777739 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:34:11,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:34:11,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00094894 s, 4.3 MB/s\n'; <rc> = 0
>Thread-14788::WARNING::2013-03-18 00:34:15,244::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:34:15,248::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:34:15,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:34:15,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163264 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:34:21,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:34:21,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826244 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:34:25,250::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:34:25,251::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:34:25,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:34:25,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122092 s, 335 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:34:25,724::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:34:25,730::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:34:25,731::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:34:31,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:34:31,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146573 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:34:35,455::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:34:35,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164792 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:34:41,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:34:41,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885444 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:34:45,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:34:45,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166374 s, 246 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:34:51,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:34:51,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014518 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:34:55,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:34:55,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124116 s, 330 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:35:01,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:35:01,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000684865 s, 6.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:35:05,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:35:05,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139848 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:35:11,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:35:11,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148455 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:35:15,676::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 00:35:15,680::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 00:35:15,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:35:15,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00976533 s, 210 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:35:15,718::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00976533 s, 210 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 00:35:15,719::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 00:35:15,719::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 00:35:15,721::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 00:35:15,723::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:35:15,723::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:35:15,724::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:35:15,724::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:35:15,724::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:35:15,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:35:16,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:35:16,140::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:35:16,141::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 00:35:16,144::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 00:35:16,145::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 00:35:16,145::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 00:35:16,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:35:16,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:35:16,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:35:16,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166253 s, 246 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:35:21,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:35:21,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011018 s, 3.7 MB/s\n'; <rc> = 0
>Thread-14789::WARNING::2013-03-18 00:35:25,757::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:35:25,761::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:35:26,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:35:26,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000960449 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:35:31,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:35:32,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863399 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:35:35,763::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:35:35,764::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:35:36,189::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:35:36,195::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:35:36,197::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:35:36,605::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:35:36,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00534191 s, 767 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:35:42,045::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:35:42,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810121 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:35:46,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:35:46,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148935 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:35:52,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:35:52,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761195 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:35:56,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:35:56,734::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00875477 s, 468 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:36:02,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:36:02,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827539 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:36:06,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:36:06,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135156 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:36:12,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:36:12,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132453 s, 309 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:36:16,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:36:16,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00541943 s, 756 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:36:22,324::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:36:22,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136603 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:36:26,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:36:26,881::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153891 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:36:32,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:36:32,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00239533 s, 1.7 MB/s\n'; <rc> = 0
>Thread-14790::WARNING::2013-03-18 00:36:36,223::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:36:36,226::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:36:36,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:36:36,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856293 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:36:42,482::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:36:42,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00652994 s, 627 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:36:46,229::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:36:46,230::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:36:46,651::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:36:46,660::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:36:46,662::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:36:46,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:36:46,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158386 s, 259 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:36:52,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:36:52,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000944852 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:36:56,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:36:57,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146713 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:37:02,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:37:02,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000658404 s, 6.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:37:07,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:37:07,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134516 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:37:12,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:37:12,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827474 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:37:17,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:37:17,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153801 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:37:22,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:37:22,809::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743813 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:37:27,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:37:27,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135596 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:37:32,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:37:32,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815613 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:37:37,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:37:37,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128197 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:37:42,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:37:42,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153273 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14791::WARNING::2013-03-18 00:37:46,690::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:37:46,693::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:37:47,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:37:47,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827135 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:37:53,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:37:53,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855246 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:37:56,694::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 00:37:56,696::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:37:56,697::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:37:57,165::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:37:57,171::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:37:57,173::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:37:57,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:37:57,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151535 s, 270 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:38:03,092::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:38:03,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832964 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:38:07,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:38:07,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140514 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:38:13,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:38:13,195::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846696 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:38:17,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:38:17,395::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858609 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:38:23,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:38:23,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873459 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:38:27,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:38:27,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144537 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:38:33,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:38:33,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778838 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:38:37,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:38:37,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159592 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:38:43,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:38:43,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00230449 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:38:47,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:38:47,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000733277 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:38:53,462::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 00:38:53,464::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 00:38:53,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:38:53,491::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00120953 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:38:53,493::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00120953 s, 1.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 00:38:53,493::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 00:38:53,493::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 00:38:53,495::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 00:38:53,496::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:38:53,497::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:38:53,497::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:38:53,498::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:38:53,498::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:38:53,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:38:53,950::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:38:53,958::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:38:53,972::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 00:38:53,987::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 00:38:53,988::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 00:38:53,988::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 00:38:53,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:38:54,347::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:38:54,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:38:54,382::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00834111 s, 491 kB/s\n'; <rc> = 0
>Thread-14792::WARNING::2013-03-18 00:38:57,202::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:38:57,207::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:38:57,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:38:57,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138629 s, 295 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:39:04,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:39:04,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131152 s, 312 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:39:07,209::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:39:07,210::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:39:07,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:39:07,618::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:39:07,625::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:39:07,626::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:39:07,657::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172599 s, 237 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:39:14,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:39:14,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000990538 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:39:17,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:39:17,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083242 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:39:24,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:39:24,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000989789 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:39:27,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:39:27,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141888 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:39:34,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:39:34,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137028 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:39:37,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:39:37,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131532 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:39:44,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:39:44,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140485 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:39:47,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:39:47,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0208717 s, 196 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:39:54,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:39:54,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158289 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:39:57,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:39:57,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137729 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:40:04,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:40:04,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912664 s, 4.5 MB/s\n'; <rc> = 0
>Thread-14793::WARNING::2013-03-18 00:40:07,676::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:40:07,682::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:40:07,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:40:07,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832382 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:40:14,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:40:14,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917542 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:40:17,685::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:40:17,686::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:40:17,944::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 00:40:17,947::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 00:40:17,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:40:18,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0149517 s, 137 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:40:18,007::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0149517 s, 137 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 00:40:18,007::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 00:40:18,007::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 00:40:18,009::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 00:40:18,010::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm reload operation' is holding the operation mutex, waiting...
>Thread-21::DEBUG::2013-03-18 00:40:18,111::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:40:18,117::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:40:18,118::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:40:18,126::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:40:18,130::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:40:18,130::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:40:18,131::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:40:18,131::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:40:18,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:40:18,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:40:18,554::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:40:18,555::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 00:40:18,558::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 00:40:18,559::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 00:40:18,559::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 00:40:18,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:40:18,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:40:18,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:40:18,958::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150152 s, 273 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:40:25,034::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:40:25,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000954362 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:40:28,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:40:29,017::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121906 s, 336 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:40:35,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:40:35,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084318 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:40:39,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:40:39,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789031 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:40:45,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:40:45,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888784 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:40:49,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:40:49,081::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781807 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:40:55,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:40:55,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00190719 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:40:59,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:40:59,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897417 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:41:05,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:41:05,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141273 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:41:09,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:41:09,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142336 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:41:15,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:41:15,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000990998 s, 4.1 MB/s\n'; <rc> = 0
>Thread-14794::WARNING::2013-03-18 00:41:18,210::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:41:18,212::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:41:19,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:41:19,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820194 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:41:25,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:41:25,526::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834764 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:41:28,215::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:41:28,216::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:41:28,656::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:41:28,662::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:41:28,664::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:41:29,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:41:29,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00950683 s, 431 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:41:35,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:41:35,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137941 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:41:39,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:41:39,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001447 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:41:45,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:41:45,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743237 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:41:49,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:41:49,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881427 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:41:55,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:41:55,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117249 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:41:59,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:41:59,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138825 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:42:05,797::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:42:05,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000697886 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:42:09,395::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:42:09,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157922 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:42:15,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:42:15,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863803 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:42:19,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:42:19,483::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825723 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:42:25,941::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:42:25,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850746 s, 4.8 MB/s\n'; <rc> = 0
>Thread-14795::WARNING::2013-03-18 00:42:28,687::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:42:28,691::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:42:29,490::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:42:29,517::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000974829 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:42:36,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:42:36,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768738 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:42:38,693::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:42:38,695::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:42:39,110::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:42:39,116::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:42:39,118::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:42:39,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:42:39,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101985 s, 402 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:42:46,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:42:46,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858685 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:42:49,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:42:49,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772366 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:42:56,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:42:56,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790006 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:42:59,612::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:42:59,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148181 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:43:06,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:43:06,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00244642 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:43:09,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:43:09,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014108 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:43:16,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:43:16,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821319 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:43:19,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:43:19,738::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882803 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:43:26,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:43:26,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801595 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:43:29,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:43:29,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017339 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:43:36,490::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:43:36,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00245903 s, 1.7 MB/s\n'; <rc> = 0
>Thread-14796::WARNING::2013-03-18 00:43:39,146::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:43:39,148::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:43:39,792::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:43:39,818::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000972284 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:43:46,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:43:46,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108255 s, 3.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:43:49,150::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 00:43:49,151::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:43:49,152::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:43:49,581::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:43:49,587::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:43:49,589::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:43:49,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:43:49,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189949 s, 216 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:43:56,647::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 00:43:56,648::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 00:43:56,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:43:56,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000875858 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:43:56,678::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000875858 s, 2.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 00:43:56,679::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 00:43:56,679::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 00:43:56,681::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 00:43:56,683::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:43:56,683::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:43:56,684::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:43:56,684::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:43:56,685::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:43:56,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:43:57,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:43:57,077::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:43:57,078::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 00:43:57,080::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 00:43:57,081::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 00:43:57,082::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 00:43:57,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:43:57,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:43:57,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:43:57,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114123 s, 359 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:43:59,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:43:59,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114962 s, 356 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:44:07,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:44:07,622::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172652 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:44:09,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:44:09,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144653 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:44:17,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:44:17,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857665 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:44:19,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:44:20,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000946754 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:44:27,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:44:27,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162849 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:44:30,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:44:30,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822262 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:44:37,814::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:44:37,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133758 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:44:40,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:44:40,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00313206 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:44:47,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:44:48,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895421 s, 4.6 MB/s\n'; <rc> = 0
>Thread-14797::WARNING::2013-03-18 00:44:49,617::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:44:49,621::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:44:50,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:44:50,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00531187 s, 771 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:44:58,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:44:58,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103799 s, 3.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:44:59,623::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:44:59,624::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:45:00,062::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:45:00,069::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:45:00,071::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:45:00,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:45:00,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123021 s, 333 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:45:08,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:45:08,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840568 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:45:10,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:45:10,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135871 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:45:18,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:45:18,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00623268 s, 657 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:45:20,237::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 00:45:20,238::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 00:45:20,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:45:20,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000993906 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:45:20,270::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000993906 s, 2.1 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 00:45:20,271::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 00:45:20,271::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 00:45:20,273::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 00:45:20,274::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:45:20,275::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:45:20,276::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:45:20,276::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:45:20,277::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:45:20,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:45:20,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:45:20,714::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:45:20,715::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 00:45:20,718::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 00:45:20,719::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 00:45:20,720::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 00:45:20,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:45:21,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:45:21,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:45:21,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848077 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:45:28,455::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:45:28,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139296 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:45:31,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:45:31,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140216 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:45:38,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:45:38,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000726804 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:45:41,156::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:45:41,188::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000698298 s, 5.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:45:48,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:45:48,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820181 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:45:51,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:45:51,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138935 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:45:58,685::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:45:58,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127536 s, 3.2 MB/s\n'; <rc> = 0
>Thread-14798::WARNING::2013-03-18 00:46:00,097::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:46:00,101::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:46:01,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:46:01,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117624 s, 348 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:46:08,899::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:46:08,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760797 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:46:10,104::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:46:10,106::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:46:10,528::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:46:10,536::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:46:10,538::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:46:11,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:46:11,399::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103223 s, 397 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:46:18,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:46:18,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000993227 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:46:21,405::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:46:21,448::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155864 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:46:29,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:46:29,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814683 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:46:31,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:46:31,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00094247 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:46:39,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:46:39,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748707 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:46:41,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:46:41,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:46:49,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:46:49,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0026697 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:46:51,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:46:51,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160012 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:46:59,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:46:59,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815893 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:47:01,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:47:01,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881067 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:47:09,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:47:09,398::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882977 s, 4.6 MB/s\n'; <rc> = 0
>Thread-14799::WARNING::2013-03-18 00:47:10,573::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:47:10,577::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:47:11,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:47:11,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795316 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:47:19,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:47:19,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816217 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:47:20,579::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:47:20,581::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:47:20,999::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:47:21,006::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:47:21,007::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:47:21,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:47:21,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126603 s, 324 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:47:29,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:47:29,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892677 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:47:31,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:47:31,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862514 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:47:39,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:47:39,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00728785 s, 562 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:47:41,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:47:41,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00192079 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:47:49,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:47:50,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907231 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:47:51,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:47:51,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150211 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:48:00,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:48:00,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00190573 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:48:01,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:48:01,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136909 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:48:10,117::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:48:10,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135221 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:48:11,875::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:48:11,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000970602 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:48:20,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:48:20,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000706581 s, 5.8 MB/s\n'; <rc> = 0
>Thread-14800::WARNING::2013-03-18 00:48:21,032::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:48:21,039::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:48:21,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:48:21,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083191 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:48:30,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:48:30,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874327 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:48:31,042::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:48:31,044::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:48:31,489::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:48:31,496::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:48:31,497::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:48:31,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:48:31,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143312 s, 286 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:48:40,612::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:48:40,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00354861 s, 1.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:48:42,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:48:42,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847565 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:48:50,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:48:50,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802882 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:48:52,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:48:52,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171405 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:49:00,768::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 00:49:00,770::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 00:49:00,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:49:00,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00527358 s, 388 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:49:00,808::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00527358 s, 388 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 00:49:00,809::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 00:49:00,809::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 00:49:00,811::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 00:49:00,813::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:49:00,813::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:49:00,814::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:49:00,814::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:49:00,815::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:49:00,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:49:01,260::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:49:01,265::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:49:01,266::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 00:49:01,268::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 00:49:01,269::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 00:49:01,270::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 00:49:01,271::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:49:01,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:49:01,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:49:01,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00696095 s, 588 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:49:02,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:49:02,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165902 s, 247 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:49:11,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:49:11,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012206 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:49:12,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:49:12,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791212 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:49:21,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:49:21,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136559 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:49:22,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:49:22,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130453 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14801::WARNING::2013-03-18 00:49:31,524::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:49:31,527::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 00:49:31,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:49:31,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816009 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:49:32,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:49:32,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091277 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:49:41,529::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 00:49:41,530::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:49:41,531::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:49:41,941::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-20::DEBUG::2013-03-18 00:49:41,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-21::WARNING::2013-03-18 00:49:41,995::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:49:42,017::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:49:42,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149099 s, 275 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:49:42,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:49:42,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112216 s, 365 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:49:52,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:49:52,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755109 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:49:52,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:49:52,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134081 s, 305 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:50:02,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:50:02,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000991782 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:50:02,345::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:50:02,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00097631 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:50:12,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:50:12,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000735867 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:50:12,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:50:12,409::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814873 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:50:22,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:50:22,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075828 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:50:22,413::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 00:50:22,415::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 00:50:22,416::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:50:22,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00125738 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:50:22,453::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00125738 s, 1.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 00:50:22,453::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 00:50:22,454::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 00:50:22,455::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 00:50:22,456::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:50:22,457::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:50:22,457::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:50:22,458::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:50:22,458::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:50:22,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:50:22,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:50:22,905::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:50:22,906::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 00:50:22,909::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 00:50:22,911::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 00:50:22,911::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 00:50:22,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:50:23,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:50:23,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:50:23,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182373 s, 225 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:50:32,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:50:32,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823894 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:50:33,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:50:33,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00218767 s, 1.9 MB/s\n'; <rc> = 0
>Thread-14802::WARNING::2013-03-18 00:50:42,073::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:50:42,079::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 00:50:42,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:50:42,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149458 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:50:43,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:50:43,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103436 s, 4.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:50:52,082::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:50:52,084::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:50:52,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:50:52,568::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:50:52,575::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:50:52,576::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:50:52,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0225759 s, 181 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:50:53,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:50:53,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135463 s, 302 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:51:02,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:51:02,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129992 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:51:03,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:51:03,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762027 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:51:12,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:51:12,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138949 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:51:13,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:51:13,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150157 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:51:22,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:51:22,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925691 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:51:23,556::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:51:23,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855796 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:51:32,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:51:32,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143555 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:51:33,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:51:33,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150707 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:51:42,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:51:43,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141555 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:51:43,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:51:43,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013727 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14803::WARNING::2013-03-18 00:51:52,626::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:51:52,632::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 00:51:53,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:51:53,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872459 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:51:53,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:51:53,709::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777041 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:52:02,634::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:52:02,635::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:52:03,062::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:52:03,069::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:52:03,070::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:52:03,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:52:03,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00919191 s, 446 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:52:03,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:52:03,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159944 s, 256 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:52:13,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:52:13,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140362 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:52:13,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:52:13,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176533 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:52:23,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:52:23,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806699 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:52:23,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:52:23,843::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167245 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:52:33,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:52:33,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906461 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:52:33,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:52:33,890::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139606 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:52:43,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:52:43,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136279 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:52:43,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:52:43,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00376521 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:52:53,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:52:53,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087149 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:52:53,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:52:53,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142452 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14804::WARNING::2013-03-18 00:53:03,099::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:53:03,105::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 00:53:03,625::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:53:03,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078549 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:53:03,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:53:04,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109536 s, 374 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:53:13,110::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:53:13,111::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:53:13,572::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:53:13,579::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:53:13,580::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:53:13,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:53:13,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178365 s, 230 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:53:14,034::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:53:14,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108837 s, 376 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:53:23,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:53:23,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824069 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:53:24,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:53:24,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0097153 s, 422 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:53:33,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:53:33,907::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000717541 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:53:34,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:53:34,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867061 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:53:43,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:53:43,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753729 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:53:44,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:53:44,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115114 s, 356 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:53:54,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:53:54,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775445 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:53:54,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:53:54,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159373 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:54:04,090::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 00:54:04,091::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 00:54:04,092::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:54:04,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00931091 s, 220 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:54:04,137::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00931091 s, 220 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 00:54:04,137::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 00:54:04,138::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 00:54:04,140::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 00:54:04,143::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:54:04,144::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:54:04,145::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:54:04,145::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:54:04,145::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:54:04,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:54:04,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:54:04,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00640435 s, 640 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:54:04,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:54:04,612::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:54:04,613::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 00:54:04,616::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 00:54:04,616::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 00:54:04,617::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 00:54:04,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:54:04,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:54:04,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:54:05,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00773081 s, 530 kB/s\n'; <rc> = 0
>Thread-14805::WARNING::2013-03-18 00:54:13,617::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:54:13,623::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:54:14,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:54:14,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000733696 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:54:15,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:54:15,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154093 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:54:23,626::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:54:23,627::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:54:24,218::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:54:24,226::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:54:24,227::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:54:24,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:54:24,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00954595 s, 429 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:54:25,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:54:25,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190483 s, 215 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:54:34,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:54:34,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134763 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:54:35,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:54:35,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915157 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:54:44,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:54:44,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000726974 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:54:45,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:54:45,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000930943 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:54:54,482::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:54:54,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120316 s, 340 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:54:55,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:54:55,392::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789713 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:55:04,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:55:04,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0073863 s, 555 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:55:05,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:55:05,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150064 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:55:14,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:55:14,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119627 s, 342 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:55:15,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:55:15,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147515 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14806::WARNING::2013-03-18 00:55:24,252::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:55:24,256::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:55:24,758::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 00:55:24,760::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 00:55:24,760::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:55:24,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000856889 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:55:24,804::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000856889 s, 2.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 00:55:24,805::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 00:55:24,805::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 00:55:24,814::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 00:55:24,815::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:55:24,816::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:55:24,817::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:55:24,817::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:55:24,817::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 00:55:24,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:55:25,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:55:25,219::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:55:25,220::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 00:55:25,222::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 00:55:25,223::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 00:55:25,224::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 00:55:25,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:55:25,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:55:25,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:55:25,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:55:25,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873019 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:55:25,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111922 s, 3.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:55:34,258::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 00:55:34,259::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:55:34,260::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:55:34,679::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:55:34,686::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:55:34,687::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:55:35,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:55:35,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:55:35,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143566 s, 285 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:55:35,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102326 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:55:45,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:55:45,791::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836582 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:55:45,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:55:45,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811837 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:55:55,796::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:55:55,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000889165 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:55:55,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:55:55,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000955497 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:56:05,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:56:05,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728429 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:56:05,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:56:05,979::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00255064 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:56:15,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:56:15,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136962 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:56:16,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:56:16,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137497 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:56:25,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:56:25,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000921365 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:56:26,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:56:26,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152535 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14807::WARNING::2013-03-18 00:56:34,716::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:56:34,723::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:56:35,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:56:35,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000889635 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:56:36,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:56:36,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873159 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:56:44,726::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:56:44,727::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:56:45,129::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:56:45,138::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:56:45,142::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:56:45,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:56:46,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143344 s, 286 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:56:46,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:56:46,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162487 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:56:56,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:56:56,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000939827 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:56:56,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:56:56,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144462 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:57:06,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:57:06,123::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.020411 s, 201 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:57:06,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:57:06,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00804811 s, 509 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:57:16,129::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:57:16,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00198218 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:57:16,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:57:16,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142877 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:57:26,171::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:57:26,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164702 s, 249 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:57:26,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:57:26,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148305 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:57:36,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:57:36,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873068 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:57:36,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:57:36,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142433 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14808::WARNING::2013-03-18 00:57:45,166::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:57:45,169::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:57:46,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:57:46,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893091 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:57:46,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:57:46,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813773 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:57:55,171::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:57:55,173::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:57:55,592::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:57:55,598::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:57:55,600::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:57:56,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:57:56,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146326 s, 280 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:57:56,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:57:56,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171271 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:58:06,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:58:06,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161767 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:58:06,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:58:06,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156658 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:58:16,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:58:16,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013701 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:58:16,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:58:16,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877214 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:58:26,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:58:26,512::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777038 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:58:27,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:58:27,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138756 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:58:36,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:58:36,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130576 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:58:37,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:58:37,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00205899 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:58:46,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:58:46,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871183 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:58:47,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:58:47,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836975 s, 4.9 MB/s\n'; <rc> = 0
>Thread-14809::WARNING::2013-03-18 00:58:55,626::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 00:58:55,630::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 00:58:56,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:58:56,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000970307 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:58:57,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:58:57,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153297 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 00:59:05,632::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 00:59:05,633::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 00:59:06,053::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 00:59:06,059::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 00:59:06,061::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 00:59:06,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:59:06,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148836 s, 275 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:59:07,341::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 00:59:07,343::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 00:59:07,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:59:07,372::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000876342 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:59:07,374::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000876342 s, 2.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 00:59:07,375::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 00:59:07,375::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 00:59:07,378::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 00:59:07,379::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:59:07,380::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:59:07,381::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:59:07,381::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:59:07,383::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 00:59:07,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:59:07,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:59:07,803::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 00:59:07,804::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 00:59:07,806::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 00:59:07,807::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 00:59:07,807::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 00:59:07,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:59:08,163::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:59:08,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:59:08,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102893 s, 398 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:59:16,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:59:16,742::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139378 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:59:18,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:59:18,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0230514 s, 178 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:59:26,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:59:26,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101614 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:59:28,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:59:28,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820776 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:59:36,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:59:36,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169001 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:59:38,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:59:38,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817446 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:59:46,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:59:46,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137987 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:59:48,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:59:48,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153078 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 00:59:56,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 00:59:56,958::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855789 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 00:59:58,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 00:59:58,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864471 s, 4.7 MB/s\n'; <rc> = 0
>Thread-14810::WARNING::2013-03-18 01:00:06,088::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:00:06,092::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:00:07,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:00:07,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00651352 s, 629 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:00:08,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:00:08,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000964568 s, 4.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:00:16,094::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:00:16,095::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:00:16,519::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:00:16,525::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:00:16,527::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:00:17,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:00:17,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190591 s, 215 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:00:18,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:00:18,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881604 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:00:27,128::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 01:00:27,130::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 01:00:27,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:00:27,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00133807 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:00:27,169::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00133807 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 01:00:27,170::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 01:00:27,170::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 01:00:27,172::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 01:00:27,174::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:00:27,174::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:00:27,175::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:00:27,175::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:00:27,175::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:00:27,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:00:27,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:00:27,613::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:00:27,614::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 01:00:27,617::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 01:00:27,618::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 01:00:27,618::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 01:00:27,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:00:27,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:00:27,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:00:28,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018334 s, 223 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:00:28,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:00:28,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862352 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:00:38,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:00:38,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00371019 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:00:38,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:00:38,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849482 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:00:48,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:00:48,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011539 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:00:48,899::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:00:48,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140002 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:00:58,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:00:58,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00362303 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:00:58,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:00:59,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147583 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:01:08,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:01:08,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155423 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:01:09,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:01:09,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108783 s, 3.8 MB/s\n'; <rc> = 0
>Thread-14811::WARNING::2013-03-18 01:01:16,563::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:01:16,568::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:01:18,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:01:18,402::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00200932 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:01:19,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:01:19,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784103 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:01:26,570::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 01:01:26,571::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:01:26,573::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:01:27,022::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:01:27,028::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:01:27,030::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:01:28,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:01:28,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177469 s, 231 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:01:29,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:01:29,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779366 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:01:38,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:01:38,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083753 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:01:39,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:01:39,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137408 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:01:48,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:01:48,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134313 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:01:49,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:01:49,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170079 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:01:58,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:01:58,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143828 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:01:59,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:01:59,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143213 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:02:08,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:02:08,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00247815 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:02:09,522::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:02:09,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754006 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:02:18,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:02:18,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016321 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:02:19,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:02:19,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814551 s, 5.0 MB/s\n'; <rc> = 0
>Thread-14812::WARNING::2013-03-18 01:02:27,057::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:02:27,061::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:02:28,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:02:28,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100799 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:02:29,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:02:29,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164823 s, 2.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:02:37,063::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:02:37,065::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:02:37,508::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:02:37,515::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:02:37,516::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:02:38,740::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:02:38,786::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186033 s, 220 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:02:39,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:02:39,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885538 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:02:48,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:02:48,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00211191 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:02:49,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:02:49,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00186314 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:02:58,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:02:58,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110276 s, 371 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:02:59,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:02:59,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171894 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:03:08,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:03:08,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013939 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:03:09,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:03:10,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772405 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:03:18,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:03:19,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00579391 s, 707 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:03:20,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:03:20,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084823 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:03:29,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:03:29,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120156 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:03:30,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:03:30,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140042 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14813::WARNING::2013-03-18 01:03:37,540::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:03:37,548::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:03:39,048::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:03:39,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000929258 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:03:40,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:03:40,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848066 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:03:47,550::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:03:47,552::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:03:47,978::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:03:47,984::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:03:47,986::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:03:49,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:03:49,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139462 s, 294 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:03:50,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:03:50,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000699182 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:03:59,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:03:59,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859169 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:04:00,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:04:00,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799153 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:04:09,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:04:09,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167629 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:04:10,424::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 01:04:10,426::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 01:04:10,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:04:10,450::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00085602 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:04:10,452::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00085602 s, 2.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 01:04:10,452::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 01:04:10,453::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 01:04:10,454::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 01:04:10,455::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:04:10,456::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:04:10,457::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:04:10,458::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:04:10,459::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:04:10,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:04:10,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:04:10,897::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:04:10,898::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 01:04:10,900::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 01:04:10,902::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 01:04:10,902::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 01:04:10,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:04:11,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:04:11,260::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:04:11,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00676092 s, 606 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:04:19,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:04:19,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142404 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:04:21,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:04:21,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130256 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:04:29,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:04:29,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895805 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:04:31,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:04:31,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115576 s, 354 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:04:39,312::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:04:39,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797474 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:04:41,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:04:41,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132631 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14814::WARNING::2013-03-18 01:04:48,010::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:04:48,014::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:04:49,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:04:49,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0009614 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:04:51,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:04:51,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854707 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:04:58,018::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:04:58,019::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:04:58,447::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:04:58,454::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:04:58,455::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:04:59,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:04:59,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177038 s, 231 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:05:01,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:05:01,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829229 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:05:09,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:05:09,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901627 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:05:11,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:05:11,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842163 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:05:19,471::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:05:19,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141493 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:05:21,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:05:21,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00095155 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:05:29,509::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 01:05:29,510::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 01:05:29,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:05:29,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00157785 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:05:29,552::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00157785 s, 1.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 01:05:29,553::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 01:05:29,553::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 01:05:29,555::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 01:05:29,556::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:05:29,556::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:05:29,557::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:05:29,557::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:05:29,557::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:05:29,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:05:30,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:05:30,056::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:05:30,057::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 01:05:30,059::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 01:05:30,061::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 01:05:30,061::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 01:05:30,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:05:30,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:05:30,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:05:30,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167725 s, 244 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:05:31,857::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:05:31,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788184 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:05:40,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:05:40,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153736 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:05:41,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:05:41,963::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00288453 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:05:50,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:05:50,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823049 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:05:52,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:05:52,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173135 s, 2.4 MB/s\n'; <rc> = 0
>Thread-14815::WARNING::2013-03-18 01:05:58,480::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:05:58,484::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:06:00,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:06:00,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832852 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:06:02,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:06:02,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00528959 s, 774 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:06:08,486::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:06:08,487::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:06:08,953::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:06:08,960::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:06:08,961::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:06:10,592::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:06:10,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00575602 s, 712 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:06:12,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:06:12,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753948 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:06:20,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:06:20,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140039 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:06:22,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:06:22,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856742 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:06:30,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:06:30,712::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000964321 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:06:32,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:06:32,366::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845577 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:06:40,717::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:06:40,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010353 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:06:42,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:06:42,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140622 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:06:50,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:06:50,786::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818031 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:06:52,490::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:06:52,529::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139147 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:07:00,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:07:00,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105499 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:07:02,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:07:02,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880631 s, 4.7 MB/s\n'; <rc> = 0
>Thread-14816::WARNING::2013-03-18 01:07:08,990::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:07:08,993::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:07:10,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:07:10,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000945231 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:07:12,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:07:12,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862286 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:07:18,997::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 01:07:18,998::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:07:19,000::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:07:19,415::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:07:19,422::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:07:19,423::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:07:20,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:07:20,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134585 s, 304 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:07:22,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:07:22,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781482 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:07:30,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:07:30,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772279 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:07:32,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:07:32,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014675 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:07:40,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:07:40,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091238 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:07:42,857::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:07:42,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740501 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:07:50,993::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:07:51,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820408 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:07:52,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:07:52,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143786 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:08:01,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:08:01,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852931 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:08:03,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:08:03,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837581 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:08:11,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:08:11,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139523 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:08:13,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:08:13,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815786 s, 5.0 MB/s\n'; <rc> = 0
>Thread-14817::WARNING::2013-03-18 01:08:19,455::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:08:19,462::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:08:21,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:08:21,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849028 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:08:23,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:08:23,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013156 s, 3.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:08:29,465::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:08:29,466::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:08:29,877::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:08:29,883::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:08:29,885::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:08:31,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:08:31,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167032 s, 245 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:08:33,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:08:33,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156135 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:08:41,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:08:41,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00098574 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:08:43,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:08:43,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015271 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:08:51,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:08:51,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00369685 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:08:53,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:08:53,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125962 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:09:01,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:09:01,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00193108 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:09:03,455::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:09:03,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165795 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:09:11,312::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:09:11,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160313 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:09:13,541::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 01:09:13,542::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 01:09:13,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:09:13,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000823015 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:09:13,576::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000823015 s, 2.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 01:09:13,577::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 01:09:13,577::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 01:09:13,579::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 01:09:13,580::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:09:13,581::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:09:13,581::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:09:13,581::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:09:13,582::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:09:13,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:09:13,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:09:13,990::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:09:13,991::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 01:09:13,993::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 01:09:13,994::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 01:09:13,995::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 01:09:13,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:09:14,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:09:14,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:09:14,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00664179 s, 617 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:09:21,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:09:21,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000675667 s, 6.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:09:24,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:09:24,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145643 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14818::WARNING::2013-03-18 01:09:29,914::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:09:29,919::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:09:31,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:09:31,439::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907759 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:09:34,732::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:09:34,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145695 s, 2.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:09:39,922::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:09:39,923::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:09:40,349::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:09:40,356::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:09:40,358::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:09:41,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:09:41,490::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015202 s, 269 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:09:44,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:09:44,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807692 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:09:51,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:09:51,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877991 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:09:54,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:09:54,912::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157324 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:10:01,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:10:01,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0404977 s, 101 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:10:04,959::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:10:04,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000910699 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:10:11,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:10:11,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100556 s, 407 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:10:15,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:10:15,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143564 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:10:21,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:10:21,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146207 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:10:25,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:10:25,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012798 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:10:31,766::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 01:10:31,768::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 01:10:31,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:10:31,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000819885 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:10:31,800::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000819885 s, 2.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 01:10:31,801::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 01:10:31,801::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 01:10:31,802::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 01:10:31,803::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:10:31,804::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:10:31,805::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:10:31,805::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:10:31,805::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:10:31,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:10:32,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:10:32,227::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:10:32,228::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 01:10:32,230::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 01:10:32,232::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 01:10:32,232::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 01:10:32,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:10:32,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:10:32,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:10:32,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182751 s, 224 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:10:35,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:10:35,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145271 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14819::WARNING::2013-03-18 01:10:40,390::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:10:40,394::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:10:42,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:10:42,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857074 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:10:45,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:10:45,309::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157076 s, 2.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:10:50,396::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:10:50,397::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:10:50,822::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:10:50,829::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:10:50,830::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:10:52,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:10:52,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128313 s, 319 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:10:55,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:10:55,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101941 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:11:02,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:11:02,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898414 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:11:05,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:11:05,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869378 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:11:12,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:11:12,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862538 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:11:15,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:11:15,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131501 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:11:22,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:11:22,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154663 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:11:25,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:11:25,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130501 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:11:32,871::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:11:32,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131958 s, 310 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:11:35,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:11:35,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771223 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:11:42,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:11:42,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131583 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:11:45,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:11:45,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146269 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14820::WARNING::2013-03-18 01:11:50,855::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:11:50,860::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:11:52,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:11:53,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160971 s, 254 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:11:55,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:11:55,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860904 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:12:00,862::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:12:00,864::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:12:01,292::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:12:01,299::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:12:01,300::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:12:03,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:12:03,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164833 s, 248 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:12:05,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:12:05,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819326 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:12:13,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:12:13,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00446004 s, 918 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:12:15,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:12:16,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014091 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:12:23,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:12:23,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149288 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:12:26,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:12:26,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113193 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:12:33,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:12:33,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00927888 s, 441 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:12:36,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:12:36,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127057 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:12:43,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:12:43,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797314 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:12:46,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:12:46,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831768 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:12:53,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:12:53,309::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123826 s, 331 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:12:56,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:12:56,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000950691 s, 4.3 MB/s\n'; <rc> = 0
>Thread-14821::WARNING::2013-03-18 01:13:01,328::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:13:01,332::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:13:03,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:13:03,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854656 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:13:06,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:13:06,424::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794628 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:13:11,333::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 01:13:11,335::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:13:11,336::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:13:11,767::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:13:11,774::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:13:11,775::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:13:13,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:13:13,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00854556 s, 479 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:13:16,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:13:16,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148806 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:13:23,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:13:23,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728615 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:13:26,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:13:26,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837245 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:13:33,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:13:33,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835469 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:13:36,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:13:36,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883373 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:13:43,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:13:43,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769091 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:13:46,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:13:46,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142874 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:13:53,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:13:53,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163588 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:13:56,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:13:56,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916612 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:14:03,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:14:03,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150342 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:14:06,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:14:06,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132716 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14822::WARNING::2013-03-18 01:14:11,816::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:14:11,822::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:14:13,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:14:13,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777911 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:14:16,924::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 01:14:16,930::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 01:14:16,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:14:17,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0184314 s, 111 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:14:17,020::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0184314 s, 111 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 01:14:17,020::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 01:14:17,021::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 01:14:17,022::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 01:14:17,023::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:14:17,024::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:14:17,024::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:14:17,025::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:14:17,025::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:14:17,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:14:17,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:14:17,483::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:14:17,484::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 01:14:17,487::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 01:14:17,488::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 01:14:17,489::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 01:14:17,490::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:14:17,837::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:14:17,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:14:17,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100921 s, 406 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:14:21,824::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:14:21,825::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:14:22,227::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:14:22,233::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:14:22,235::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:14:23,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:14:23,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131897 s, 311 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:14:27,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:14:27,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814657 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:14:33,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:14:33,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843575 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:14:37,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:14:38,010::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810024 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:14:43,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:14:43,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850674 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:14:48,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:14:48,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887363 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:14:53,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:14:53,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775619 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:14:58,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:14:58,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144866 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:15:03,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:15:03,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130587 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:15:08,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:15:08,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141023 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:15:13,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:15:13,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161568 s, 254 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:15:18,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:15:18,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137833 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14823::WARNING::2013-03-18 01:15:22,261::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:15:22,266::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:15:23,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:15:23,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807405 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:15:28,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:15:28,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167794 s, 2.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:15:32,268::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:15:32,269::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:15:32,693::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:15:32,700::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:15:32,701::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:15:33,936::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 01:15:33,939::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 01:15:33,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:15:33,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0151799 s, 135 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:15:33,984::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0151799 s, 135 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 01:15:33,984::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 01:15:33,985::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 01:15:33,987::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 01:15:33,988::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:15:33,989::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:15:33,990::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:15:33,990::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:15:33,991::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:15:33,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:15:34,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:15:34,353::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:15:34,354::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 01:15:34,357::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 01:15:34,359::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 01:15:34,359::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 01:15:34,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:15:34,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:15:34,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:15:34,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0228012 s, 180 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:15:38,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:15:38,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870366 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:15:44,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:15:44,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000745643 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:15:48,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:15:48,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813849 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:15:54,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:15:54,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812125 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:15:58,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:15:58,627::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138564 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:16:04,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:16:04,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159607 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:16:08,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:16:08,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748501 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:16:14,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:16:14,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174556 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:16:18,738::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:16:18,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800955 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:16:24,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:16:25,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788749 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:16:28,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:16:28,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130043 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14824::WARNING::2013-03-18 01:16:32,731::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:16:32,737::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:16:35,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:16:35,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014612 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:16:38,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:16:38,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751295 s, 5.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:16:42,739::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:16:42,740::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:16:43,158::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:16:43,165::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:16:43,166::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:16:45,092::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:16:45,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0187972 s, 218 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:16:48,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:16:48,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156062 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:16:55,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:16:55,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775211 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:16:59,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:16:59,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00227767 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:17:05,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:17:05,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155581 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:17:09,117::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:17:09,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000733997 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:17:15,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:17:15,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830575 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:17:19,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:17:19,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00794901 s, 515 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:17:25,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:17:25,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129422 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:17:29,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:17:29,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076547 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:17:35,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:17:35,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016178 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:17:39,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:17:39,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000966838 s, 4.2 MB/s\n'; <rc> = 0
>Thread-14825::WARNING::2013-03-18 01:17:43,206::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:17:43,210::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:17:45,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:17:45,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000940712 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:17:49,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:17:49,456::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857691 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:17:53,212::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:17:53,213::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:17:53,631::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:17:53,638::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:17:53,640::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:17:55,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:17:55,552::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110154 s, 372 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:17:59,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:17:59,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136867 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:18:05,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:18:05,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081109 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:18:09,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:18:09,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147218 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:18:15,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:18:15,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100812 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:18:19,732::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:18:19,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832273 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:18:25,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:18:25,692::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137854 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:18:29,797::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:18:29,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153697 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:18:35,697::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:18:35,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135635 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:18:39,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:18:39,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137351 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:18:45,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:18:45,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826416 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:18:49,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:18:50,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125244 s, 3.3 MB/s\n'; <rc> = 0
>Thread-14826::WARNING::2013-03-18 01:18:53,671::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:18:53,674::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:18:55,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:18:55,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825996 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:19:00,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:19:00,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133506 s, 3.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:19:03,687::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 01:19:03,688::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:19:03,689::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:19:04,136::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:19:04,142::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:19:04,144::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:19:05,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:19:05,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126936 s, 323 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:19:10,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:19:10,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000709757 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:19:15,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:19:15,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125404 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:19:20,208::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 01:19:20,210::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 01:19:20,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:19:20,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00189386 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:19:20,257::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00189386 s, 1.1 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 01:19:20,258::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 01:19:20,258::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 01:19:20,259::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 01:19:20,260::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:19:20,261::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:19:20,261::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:19:20,262::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:19:20,262::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:19:20,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:19:20,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:19:20,710::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:19:20,711::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 01:19:20,714::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 01:19:20,715::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 01:19:20,715::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 01:19:20,717::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:19:21,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:19:21,108::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:19:21,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130106 s, 315 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:19:25,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:19:25,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182601 s, 224 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:19:31,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:19:31,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825592 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:19:35,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:19:36,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133736 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:19:41,281::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:19:41,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148842 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:19:46,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:19:46,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141419 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:19:51,373::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:19:51,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156641 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:19:56,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:19:56,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014568 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:20:01,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:20:01,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135346 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14827::WARNING::2013-03-18 01:20:04,172::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:20:04,177::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:20:06,117::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:20:06,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0157586 s, 260 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:20:11,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:20:11,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817041 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:20:14,180::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:20:14,182::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:20:14,588::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:20:14,594::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:20:14,596::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:20:16,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:20:16,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101055 s, 405 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:20:21,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:20:21,658::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000965523 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:20:26,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:20:26,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771252 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:20:31,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:20:31,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831237 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:20:36,250::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 01:20:36,252::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 01:20:36,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:20:36,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00141082 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:20:36,297::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00141082 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 01:20:36,298::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 01:20:36,298::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 01:20:36,300::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 01:20:36,301::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:20:36,301::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:20:36,302::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:20:36,302::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:20:36,303::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:20:36,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:20:36,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:20:36,736::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:20:36,737::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 01:20:36,739::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 01:20:36,740::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 01:20:36,741::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 01:20:36,742::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:20:37,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:20:37,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:20:37,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0205507 s, 199 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:20:41,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:20:41,801::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798869 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:20:47,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:20:47,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783207 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:20:51,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:20:51,880::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136733 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:20:57,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:20:57,270::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151871 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:21:01,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:21:01,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012491 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:21:07,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:21:07,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143015 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:21:12,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:21:12,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828355 s, 4.9 MB/s\n'; <rc> = 0
>Thread-14828::WARNING::2013-03-18 01:21:14,622::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:21:14,625::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:21:17,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:21:17,372::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121775 s, 336 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:21:22,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:21:22,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774753 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:21:24,627::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:21:24,628::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:21:25,059::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:21:25,065::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:21:25,067::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:21:27,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:21:27,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00474093 s, 864 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:21:32,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:21:32,191::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824803 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:21:37,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:21:37,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102675 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:21:42,230::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:21:42,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086058 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:21:47,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:21:47,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843455 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:21:52,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:21:52,350::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151366 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:21:57,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:21:57,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147018 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:22:02,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:22:02,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0018762 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:22:07,560::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:22:07,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132949 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:22:12,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:22:12,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123098 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:22:17,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:22:17,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155113 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:22:22,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:22:22,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139887 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14829::WARNING::2013-03-18 01:22:25,097::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:22:25,100::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:22:27,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:22:27,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867134 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:22:32,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:22:32,692::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850263 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:22:35,102::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:22:35,103::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:22:35,597::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:22:35,604::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:22:35,605::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:22:37,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:22:37,744::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166375 s, 246 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:22:42,732::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:22:42,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776096 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:22:47,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:22:47,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00842493 s, 486 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:22:52,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:22:52,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00724785 s, 565 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:22:57,797::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:22:57,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000929054 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:23:02,894::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:23:02,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145608 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:23:07,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:23:07,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000938065 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:23:12,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:23:13,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794275 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:23:17,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:23:17,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144827 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:23:23,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:23:23,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128744 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:23:27,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:23:27,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00886789 s, 462 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:23:33,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:23:33,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010387 s, 3.9 MB/s\n'; <rc> = 0
>Thread-14830::WARNING::2013-03-18 01:23:35,633::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:23:35,638::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:23:38,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:23:38,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091018 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:23:43,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:23:43,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762562 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:23:45,640::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:23:45,641::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:23:46,048::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:23:46,055::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:23:46,056::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:23:48,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:23:48,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150188 s, 273 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:23:53,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:23:53,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845713 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:23:58,092::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:23:58,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111308 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:24:03,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:24:03,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886419 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:24:08,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:24:08,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759035 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:24:13,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:24:13,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016034 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:24:18,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:24:18,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134308 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:24:23,503::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 01:24:23,505::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 01:24:23,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:24:23,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00125317 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:24:23,547::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00125317 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 01:24:23,548::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 01:24:23,548::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 01:24:23,550::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 01:24:23,551::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:24:23,551::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:24:23,552::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:24:23,552::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:24:23,552::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:24:23,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:24:23,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:24:23,984::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:24:23,985::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 01:24:23,987::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 01:24:23,989::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 01:24:23,989::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 01:24:23,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:24:24,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:24:24,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:24:24,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176389 s, 232 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:24:28,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:24:28,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143398 s, 286 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:24:34,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:24:34,478::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847562 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:24:38,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:24:38,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126007 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:24:44,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:24:44,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831274 s, 4.9 MB/s\n'; <rc> = 0
>Thread-14831::WARNING::2013-03-18 01:24:46,084::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:24:46,091::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:24:48,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:24:48,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773655 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:24:54,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:24:54,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728698 s, 5.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:24:56,093::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 01:24:56,094::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:24:56,095::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:24:56,568::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:24:56,575::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:24:56,576::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:24:58,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:24:58,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015271 s, 268 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:25:04,861::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:25:04,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100721 s, 407 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:25:08,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:25:08,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000956894 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:25:14,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:25:14,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858831 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:25:18,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:25:18,481::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017239 s, 238 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:25:25,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:25:25,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130133 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:25:28,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:25:28,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00205532 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:25:35,108::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:25:35,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008101 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:25:38,523::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 01:25:38,525::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 01:25:38,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:25:38,561::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00131999 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:25:38,562::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00131999 s, 1.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 01:25:38,563::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 01:25:38,563::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 01:25:38,565::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 01:25:38,566::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:25:38,567::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:25:38,567::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:25:38,567::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:25:38,568::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:25:38,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:25:39,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:25:39,029::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:25:39,039::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 01:25:39,042::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 01:25:39,043::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 01:25:39,043::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 01:25:39,045::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:25:39,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:25:39,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:25:39,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113015 s, 362 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:25:45,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:25:45,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136418 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:25:49,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:25:49,548::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899658 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:25:55,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:25:55,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0426552 s, 96.0 kB/s\n'; <rc> = 0
>Thread-14832::WARNING::2013-03-18 01:25:56,603::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:25:56,607::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:25:59,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:25:59,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826213 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:26:05,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:26:05,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828915 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:26:06,609::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:26:06,610::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:26:07,028::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:26:07,034::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:26:07,036::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:26:09,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:26:09,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160043 s, 256 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:26:15,648::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:26:15,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809111 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:26:19,648::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:26:19,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083762 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:26:25,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:26:25,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00833159 s, 492 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:26:29,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:26:29,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079467 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:26:35,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:26:35,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133024 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:26:39,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:26:39,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013314 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:26:45,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:26:45,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778751 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:26:49,760::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:26:49,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130688 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:26:55,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:26:55,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150286 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:26:59,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:26:59,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015782 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:27:06,060::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:27:06,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812463 s, 5.0 MB/s\n'; <rc> = 0
>Thread-14833::WARNING::2013-03-18 01:27:07,064::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:27:07,070::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:27:09,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:27:09,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829628 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:27:16,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:27:16,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000718019 s, 5.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:27:17,073::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:27:17,075::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:27:17,506::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:27:17,513::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:27:17,514::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:27:19,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:27:19,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014682 s, 279 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:27:26,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:27:26,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000928153 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:27:29,938::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:27:29,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092515 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:27:36,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:27:36,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135596 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:27:39,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:27:40,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793624 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:27:46,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:27:46,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124917 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:27:50,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:27:50,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077984 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:27:56,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:27:56,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877131 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:28:00,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:28:00,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875443 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:28:06,861::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:28:06,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168903 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:28:10,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:28:10,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00196335 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:28:16,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:28:17,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0029597 s, 1.4 MB/s\n'; <rc> = 0
>Thread-14834::WARNING::2013-03-18 01:28:17,539::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:28:17,543::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:28:20,118::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:28:20,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873856 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:28:27,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:28:27,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000910813 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:28:27,545::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:28:27,547::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:28:27,928::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:28:27,935::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:28:27,936::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:28:30,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:28:30,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0198364 s, 206 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:28:37,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:28:37,347::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143453 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:28:40,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:28:40,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808326 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:28:47,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:28:47,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823524 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:28:50,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:28:50,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075431 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:28:57,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:28:57,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764648 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:29:00,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:29:00,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125609 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:29:07,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:29:07,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147597 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:29:10,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:29:10,365::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086164 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:29:17,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:29:17,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165563 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:29:20,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:29:20,404::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820551 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:29:27,731::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 01:29:27,735::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 01:29:27,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:29:27,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00106271 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:29:27,806::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00106271 s, 1.9 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 01:29:27,809::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 01:29:27,811::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 01:29:27,815::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 01:29:27,822::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:29:27,827::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:29:27,829::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:29:27,831::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:29:27,834::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:29:27,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-14835::WARNING::2013-03-18 01:29:27,967::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:29:27,975::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 01:29:28,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:29:28,256::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:29:28,257::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 01:29:28,259::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 01:29:28,261::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 01:29:28,261::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 01:29:28,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:29:28,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:29:28,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:29:28,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149111 s, 275 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:29:30,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:29:30,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015661 s, 262 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:29:37,978::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:29:37,979::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:29:38,444::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:29:38,451::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:29:38,452::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:29:38,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:29:38,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132531 s, 309 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:29:40,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:29:40,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141171 s, 290 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:29:48,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:29:48,880::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860315 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:29:50,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:29:50,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104615 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:29:58,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:29:58,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00098138 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:30:00,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:30:00,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776371 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:30:08,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:30:09,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139467 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:30:10,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:30:10,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015254 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:30:19,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:30:19,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821283 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:30:20,634::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:30:20,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00857568 s, 478 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:30:29,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:30:29,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134022 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:30:30,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:30:30,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143674 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14836::WARNING::2013-03-18 01:30:38,477::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:30:38,481::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 01:30:39,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:30:39,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848837 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:30:40,734::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 01:30:40,736::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 01:30:40,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:30:40,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00133794 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:30:40,781::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00133794 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 01:30:40,782::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 01:30:40,782::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 01:30:40,784::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 01:30:40,785::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:30:40,785::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:30:40,786::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:30:40,786::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:30:40,786::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:30:40,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:30:41,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:30:41,231::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:30:41,232::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 01:30:41,235::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 01:30:41,236::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 01:30:41,236::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 01:30:41,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:30:41,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:30:41,596::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:30:41,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152618 s, 268 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:30:48,490::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 01:30:48,492::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:30:48,493::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:30:48,951::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:30:48,958::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:30:48,959::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:30:49,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:30:49,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00933761 s, 439 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:30:51,647::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:30:51,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163189 s, 251 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:30:59,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:30:59,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142186 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:31:01,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:31:01,734::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769692 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:31:09,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:31:09,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813073 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:31:11,741::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:31:11,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853277 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:31:19,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:31:19,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752525 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:31:21,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:31:21,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014982 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:31:29,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:31:29,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133426 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:31:31,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:31:31,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139502 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:31:39,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:31:39,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132361 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:31:41,857::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:31:41,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00276513 s, 1.5 MB/s\n'; <rc> = 0
>Thread-14837::WARNING::2013-03-18 01:31:48,986::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:31:48,989::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 01:31:49,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:31:49,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150179 s, 273 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:31:51,899::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:31:51,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00090363 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:31:58,993::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:31:58,994::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:31:59,479::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:31:59,486::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:31:59,487::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:31:59,854::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:31:59,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00910886 s, 450 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:32:01,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:32:01,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00860389 s, 476 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:32:09,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:32:09,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00941431 s, 435 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:32:11,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:32:12,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150025 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:32:20,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:32:20,054::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016298 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:32:22,033::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:32:22,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100975 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:32:30,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:32:30,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000940319 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:32:32,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:32:32,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014767 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:32:40,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:32:40,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101874 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:32:42,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:32:42,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00227525 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:32:50,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:32:50,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757786 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:32:52,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:32:52,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106273 s, 3.9 MB/s\n'; <rc> = 0
>Thread-14838::WARNING::2013-03-18 01:32:59,517::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:32:59,520::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 01:33:00,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:33:00,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875354 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:33:02,209::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:33:02,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904728 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:33:09,522::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:33:09,524::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:33:09,989::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:33:09,995::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:33:09,997::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:33:10,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:33:10,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00936865 s, 437 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:33:12,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:33:12,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115169 s, 356 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:33:20,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:33:20,512::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100308 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:33:22,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:33:22,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00912035 s, 449 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:33:30,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:33:30,622::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133915 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:33:32,341::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:33:32,382::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158887 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:33:40,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:33:40,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000726644 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:33:42,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:33:42,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764078 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:33:50,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:33:50,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794553 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:33:52,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:33:52,456::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000731589 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:34:00,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:34:00,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000939408 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:34:02,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:34:02,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142598 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14839::WARNING::2013-03-18 01:34:10,028::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:34:10,032::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 01:34:10,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:34:10,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740171 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:34:12,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:34:12,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862403 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:34:20,033::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:34:20,035::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:34:20,477::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:34:20,484::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:34:20,485::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:34:20,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:34:20,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870734 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:34:22,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:34:22,569::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936809 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:34:31,026::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 01:34:31,028::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 01:34:31,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:34:31,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000818696 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:34:31,054::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000818696 s, 2.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 01:34:31,055::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 01:34:31,055::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 01:34:31,057::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 01:34:31,061::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:34:31,061::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:34:31,062::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:34:31,062::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:34:31,063::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:34:31,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:34:31,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:34:31,505::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:34:31,507::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 01:34:31,509::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 01:34:31,510::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 01:34:31,511::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 01:34:31,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:34:31,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:34:31,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:34:31,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00738483 s, 555 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:34:32,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:34:32,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174731 s, 234 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:34:41,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:34:41,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761653 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:34:42,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:34:42,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836833 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:34:52,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:34:52,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000736419 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:34:52,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:34:52,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144583 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:35:02,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:35:02,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842151 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:35:02,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:35:02,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083566 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:35:12,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:35:12,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143752 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:35:12,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:35:12,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154722 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14840::WARNING::2013-03-18 01:35:20,520::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:35:20,522::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 01:35:22,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:35:22,270::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784725 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:35:22,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:35:22,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00194004 s, 2.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:35:30,528::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:35:30,529::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:35:31,011::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:35:31,018::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:35:31,020::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:35:32,312::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:35:32,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158975 s, 258 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:35:32,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:35:32,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153151 s, 267 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:35:42,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:35:42,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834553 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:35:42,926::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 01:35:42,931::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 01:35:42,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:35:42,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00077358 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:35:42,972::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00077358 s, 2.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 01:35:42,972::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 01:35:42,973::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 01:35:42,974::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 01:35:42,975::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:35:42,976::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:35:42,976::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:35:42,976::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:35:42,977::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:35:42,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:35:43,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:35:43,416::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:35:43,417::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 01:35:43,420::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 01:35:43,421::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 01:35:43,421::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 01:35:43,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:35:43,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:35:43,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:35:43,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108495 s, 378 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:35:52,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:35:52,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853558 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:35:53,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:35:53,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000940127 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:36:02,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:36:02,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827103 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:36:03,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:36:03,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899893 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:36:12,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:36:12,633::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087159 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:36:13,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:36:13,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141733 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:36:22,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:36:22,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137033 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:36:23,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:36:23,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798082 s, 5.1 MB/s\n'; <rc> = 0
>Thread-14841::WARNING::2013-03-18 01:36:31,047::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:36:31,052::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 01:36:32,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:36:32,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000938099 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:36:34,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:36:34,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171857 s, 2.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:36:41,055::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 01:36:41,056::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:36:41,057::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:36:41,652::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:36:41,658::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:36:41,659::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:36:42,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:36:42,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00966015 s, 424 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:36:44,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:36:44,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0219334 s, 187 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:36:52,899::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:36:52,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923321 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:36:54,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:36:54,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136379 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:37:02,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:37:03,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154823 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:37:04,156::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:37:04,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139895 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:37:13,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:37:13,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852429 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:37:14,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:37:14,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165564 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:37:23,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:37:23,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932953 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:37:24,227::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:37:24,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851289 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:37:33,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:37:33,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737098 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:37:34,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:37:34,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145732 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14842::WARNING::2013-03-18 01:37:41,684::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:37:41,689::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 01:37:43,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:37:43,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000979315 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:37:44,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:37:44,350::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111124 s, 369 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:37:51,691::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:37:51,692::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:37:52,138::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:37:52,145::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:37:52,146::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:37:53,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:37:53,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113597 s, 361 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:37:54,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:37:54,402::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184422 s, 222 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:38:03,412::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:38:03,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813724 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:38:04,412::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:38:04,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160321 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:38:13,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:38:13,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886792 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:38:14,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:38:14,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900798 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:38:23,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:38:23,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126691 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:38:24,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:38:24,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139288 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:38:33,625::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:38:33,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835765 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:38:34,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:38:34,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150879 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:38:43,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:38:43,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785085 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:38:44,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:38:44,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813576 s, 5.0 MB/s\n'; <rc> = 0
>Thread-14843::WARNING::2013-03-18 01:38:52,176::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:38:52,180::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 01:38:53,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:38:53,793::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771304 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:38:54,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:38:54,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014747 s, 278 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:39:02,181::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:39:02,183::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:39:02,606::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:39:02,613::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:39:02,615::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:39:03,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:39:03,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142716 s, 287 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:39:04,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:39:04,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133102 s, 308 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:39:13,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:39:13,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192303 s, 213 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:39:14,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:39:14,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152922 s, 268 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:39:24,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:39:24,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139793 s, 293 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:39:24,797::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:39:24,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000970505 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:39:34,095::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 01:39:34,096::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 01:39:34,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:39:34,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000686085 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:39:34,130::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000686085 s, 3.0 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 01:39:34,131::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 01:39:34,131::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 01:39:34,133::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 01:39:34,134::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:39:34,135::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:39:34,135::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:39:34,135::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:39:34,136::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:39:34,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:39:34,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:39:34,586::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:39:34,587::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 01:39:34,589::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 01:39:34,590::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 01:39:34,591::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 01:39:34,592::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:39:34,837::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:39:34,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00970264 s, 422 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:39:34,979::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:39:34,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:39:35,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117083 s, 350 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:39:44,899::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:39:44,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135409 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:39:45,060::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:39:45,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840366 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:39:54,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:39:54,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075648 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:39:55,129::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:39:55,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089274 s, 4.6 MB/s\n'; <rc> = 0
>Thread-14844::WARNING::2013-03-18 01:40:02,642::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:40:02,648::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:40:04,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:40:05,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839034 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:40:05,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:40:05,236::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000895598 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:40:12,650::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:40:12,651::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:40:13,112::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:40:13,119::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:40:13,120::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:40:15,019::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:40:15,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00575351 s, 712 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:40:15,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:40:15,331::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120529 s, 340 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:40:25,070::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:40:25,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152122 s, 269 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:40:25,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:40:25,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838749 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:40:35,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:40:35,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150094 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:40:35,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:40:35,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764573 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:40:45,166::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 01:40:45,167::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 01:40:45,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:40:45,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000771076 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:40:45,198::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000771076 s, 2.7 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 01:40:45,199::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 01:40:45,199::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 01:40:45,201::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 01:40:45,202::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:40:45,202::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:40:45,203::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:40:45,203::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:40:45,203::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:40:45,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:40:45,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:40:45,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000616554 s, 6.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:40:45,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:40:45,651::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:40:45,651::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 01:40:45,653::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 01:40:45,655::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 01:40:45,655::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 01:40:45,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:40:46,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:40:46,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:40:46,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0076643 s, 534 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:40:55,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:40:55,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168288 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:40:56,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:40:56,132::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117297 s, 349 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:41:05,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:41:05,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179368 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:41:06,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:41:06,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791719 s, 5.2 MB/s\n'; <rc> = 0
>Thread-14845::WARNING::2013-03-18 01:41:13,148::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:41:13,154::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 01:41:15,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:41:15,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158256 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:41:16,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:41:16,222::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00815081 s, 503 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:41:23,157::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:41:23,158::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:41:23,559::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:41:23,566::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:41:23,567::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:41:25,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:41:25,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000709927 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:41:26,227::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:41:26,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115826 s, 354 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:41:35,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:41:35,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146256 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:41:36,271::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:41:36,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00094292 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:41:45,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:41:46,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000973366 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:41:46,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:41:46,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014082 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:41:56,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:41:56,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140263 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:41:56,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:41:56,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104352 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:42:06,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:42:06,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754986 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:42:06,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:42:06,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833511 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:42:16,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:42:16,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103203 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:42:16,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:42:16,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106228 s, 3.9 MB/s\n'; <rc> = 0
>Thread-14846::WARNING::2013-03-18 01:42:23,599::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:42:23,604::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 01:42:26,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:42:26,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809084 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:42:26,471::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:42:26,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904169 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:42:33,606::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 01:42:33,608::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:42:33,609::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:42:34,036::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:42:34,042::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:42:34,044::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:42:36,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:42:36,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847692 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:42:36,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:42:36,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164939 s, 248 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:42:46,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:42:46,494::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909254 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:42:46,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:42:46,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873972 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:42:56,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:42:56,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:42:56,611::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165657 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:42:56,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849783 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:43:06,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:43:06,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:43:06,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00835107 s, 490 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:43:06,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163798 s, 250 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:43:16,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:43:16,756::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134497 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:43:16,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:43:16,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139918 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:43:26,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:43:26,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753694 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:43:26,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:43:26,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133981 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14847::WARNING::2013-03-18 01:43:34,070::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:43:34,074::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:43:36,796::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:43:36,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932383 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:43:36,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:43:36,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796317 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:43:44,076::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:43:44,077::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:43:44,511::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:43:44,518::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:43:44,519::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:43:46,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:43:46,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153981 s, 266 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:43:46,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:43:47,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843854 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:43:56,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:43:56,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152848 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:43:57,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:43:57,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144083 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:44:06,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:44:06,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000987654 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:44:07,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:44:07,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860281 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:44:16,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:44:16,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100002 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:44:17,221::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:44:17,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0024318 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:44:26,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:44:27,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156545 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:44:27,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:44:27,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127698 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:44:37,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:44:37,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132909 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:44:37,375::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 01:44:37,376::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 01:44:37,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:44:37,413::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00124813 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:44:37,415::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00124813 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 01:44:37,415::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 01:44:37,415::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 01:44:37,417::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 01:44:37,419::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:44:37,420::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:44:37,420::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:44:37,420::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:44:37,421::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:44:37,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:44:37,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:44:37,811::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:44:37,812::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 01:44:37,815::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 01:44:37,816::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 01:44:37,816::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 01:44:37,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:44:38,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:44:38,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:44:38,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849507 s, 4.8 MB/s\n'; <rc> = 0
>Thread-14848::WARNING::2013-03-18 01:44:44,546::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:44:44,553::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:44:47,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:44:47,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000910491 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:44:48,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:44:48,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842757 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:44:54,557::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:44:54,558::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:44:54,992::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:44:54,999::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:44:55,000::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:44:57,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:44:57,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178451 s, 230 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:44:58,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:44:58,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769623 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:45:07,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:45:07,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907347 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:45:08,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:45:08,409::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818064 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:45:17,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:45:17,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851244 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:45:18,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:45:18,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791894 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:45:27,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:45:27,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160643 s, 255 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:45:28,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:45:28,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130589 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:45:37,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:45:37,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185966 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:45:38,603::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:45:38,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165389 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:45:47,326::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 01:45:47,330::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 01:45:47,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:45:47,366::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00361521 s, 566 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:45:47,368::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00361521 s, 566 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 01:45:47,368::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 01:45:47,369::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 01:45:47,371::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 01:45:47,372::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:45:47,373::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:45:47,374::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:45:47,374::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:45:47,374::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:45:47,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:45:47,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:45:47,810::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:45:47,811::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 01:45:47,813::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 01:45:47,814::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 01:45:47,815::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 01:45:47,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:45:48,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:45:48,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:45:48,222::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00270307 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:45:48,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:45:48,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010893 s, 376 kB/s\n'; <rc> = 0
>Thread-14849::WARNING::2013-03-18 01:45:55,032::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:45:55,035::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:45:58,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:45:58,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000722183 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:45:58,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:45:58,809::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00959747 s, 427 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:46:05,037::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:46:05,038::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:46:05,463::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:46:05,470::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:46:05,471::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:46:08,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:46:08,311::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177616 s, 231 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:46:08,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:46:08,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801862 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:46:18,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:46:18,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000966351 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:46:18,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:46:18,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861336 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:46:28,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:46:28,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000714808 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:46:28,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:46:29,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010218 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:46:38,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:46:38,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134735 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:46:39,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:46:39,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106506 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:46:48,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:46:48,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146043 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:46:49,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:46:49,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140193 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:46:58,481::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:46:58,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010701 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:46:59,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:46:59,287::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866112 s, 4.7 MB/s\n'; <rc> = 0
>Thread-14850::WARNING::2013-03-18 01:47:05,498::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:47:05,502::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:47:08,524::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:47:08,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872106 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:47:09,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:47:09,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142114 s, 2.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:47:15,504::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:47:15,505::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:47:15,985::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:47:15,992::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:47:15,993::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:47:18,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:47:18,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189988 s, 216 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:47:19,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:47:19,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000922458 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:47:28,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:47:28,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814982 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:47:29,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:47:29,521::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137516 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:47:38,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:47:38,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788397 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:47:39,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:47:39,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797819 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:47:48,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:47:48,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001315 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:47:49,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:47:49,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145744 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:47:58,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:47:58,766::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00309763 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:47:59,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:47:59,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133875 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:48:08,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:48:08,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00240703 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:48:09,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:48:09,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00227921 s, 1.8 MB/s\n'; <rc> = 0
>Thread-14851::WARNING::2013-03-18 01:48:16,022::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:48:16,028::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:48:18,814::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:48:18,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150466 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:48:19,871::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:48:19,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907422 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:48:26,030::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 01:48:26,031::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:48:26,031::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:48:26,497::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:48:26,504::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:48:26,505::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:48:28,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:48:28,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184209 s, 222 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:48:29,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:48:29,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865133 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:48:38,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:48:38,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801586 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:48:40,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:48:40,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823597 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:48:48,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:48:48,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121897 s, 336 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:48:50,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:48:50,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127478 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:48:58,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:48:59,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837041 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:49:00,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:49:00,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000945418 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:49:09,033::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:49:09,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103636 s, 395 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:49:10,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:49:10,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150093 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:49:19,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:49:19,130::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123334 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:49:20,305::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:49:20,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841226 s, 4.9 MB/s\n'; <rc> = 0
>Thread-14852::WARNING::2013-03-18 01:49:26,534::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:49:26,541::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:49:29,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:49:29,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156662 s, 261 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:49:30,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:49:30,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000959307 s, 4.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:49:36,542::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:49:36,544::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:49:36,967::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:49:36,974::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:49:36,975::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:49:39,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:49:39,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178414 s, 230 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:49:40,466::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 01:49:40,467::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 01:49:40,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:49:40,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00072265 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:49:40,501::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00072265 s, 2.8 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 01:49:40,502::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 01:49:40,502::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 01:49:40,504::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 01:49:40,505::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:49:40,506::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:49:40,507::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:49:40,507::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:49:40,508::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:49:40,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:49:40,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:49:40,933::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:49:40,934::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 01:49:40,937::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 01:49:40,938::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 01:49:40,938::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 01:49:40,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:49:41,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:49:41,292::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:49:41,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00567165 s, 722 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:49:49,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:49:49,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866638 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:49:51,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:49:51,409::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135426 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:49:59,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:49:59,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740417 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:50:01,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:50:01,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771659 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:50:09,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:50:09,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00228073 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:50:11,605::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:50:11,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760227 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:50:19,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:50:19,424::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799699 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:50:21,681::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:50:21,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859476 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:50:29,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:50:29,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753054 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:50:31,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:50:31,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000929906 s, 4.4 MB/s\n'; <rc> = 0
>Thread-14853::WARNING::2013-03-18 01:50:37,001::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:50:37,007::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:50:39,524::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:50:39,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000984161 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:50:41,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:50:41,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818609 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:50:47,008::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:50:47,010::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:50:47,438::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:50:47,445::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:50:47,446::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:50:49,560::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 01:50:49,564::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 01:50:49,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:50:49,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00884601 s, 232 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:50:49,608::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00884601 s, 232 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 01:50:49,609::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 01:50:49,609::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 01:50:49,613::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 01:50:49,614::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:50:49,614::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:50:49,615::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:50:49,615::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:50:49,615::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:50:49,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:50:50,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:50:50,042::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:50:50,043::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 01:50:50,046::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 01:50:50,047::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 01:50:50,047::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 01:50:50,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:50:50,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:50:50,445::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:50:50,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137675 s, 298 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:50:51,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:50:51,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760095 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:51:00,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:51:00,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000682072 s, 6.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:51:01,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:51:02,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131586 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:51:10,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:51:10,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00395133 s, 1.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:51:12,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:51:12,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132057 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:51:20,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:51:20,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0033869 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:51:22,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:51:22,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00821138 s, 499 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:51:30,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:51:30,765::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00317769 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:51:32,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:51:32,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139274 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:51:40,778::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:51:40,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117838 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:51:42,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:51:42,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763955 s, 5.4 MB/s\n'; <rc> = 0
>Thread-14854::WARNING::2013-03-18 01:51:47,471::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:51:47,475::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:51:50,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:51:50,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000996274 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:51:52,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:51:52,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857728 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:51:57,477::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:51:57,478::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:51:57,953::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:51:57,960::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:51:57,962::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:52:00,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:52:00,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013608 s, 301 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:52:02,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:52:02,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0303765 s, 135 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:52:10,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:52:10,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151036 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:52:12,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:52:12,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773482 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:52:20,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:52:21,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841067 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:52:22,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:52:22,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148149 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:52:31,033::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:52:31,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079771 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:52:32,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:52:32,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793462 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:52:41,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:52:41,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763584 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:52:42,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:52:42,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844164 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:52:51,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:52:51,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762047 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:52:52,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:52:52,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793297 s, 5.2 MB/s\n'; <rc> = 0
>Thread-14855::WARNING::2013-03-18 01:52:57,992::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:52:57,998::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:53:01,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:53:01,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799983 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:53:02,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:53:03,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849463 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:53:08,000::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:53:08,001::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:53:08,476::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:53:08,482::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:53:08,483::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:53:11,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:53:11,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174947 s, 234 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:53:13,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:53:13,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156943 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:53:21,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:53:21,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843824 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:53:23,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:53:23,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166724 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:53:31,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:53:31,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840403 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:53:33,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:53:33,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846965 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:53:41,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:53:41,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878366 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:53:43,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:53:43,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142706 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:53:51,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:53:51,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125809 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:53:53,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:53:53,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013365 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:54:01,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:54:01,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00309179 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:54:03,455::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:54:03,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126 s, 3.3 MB/s\n'; <rc> = 0
>Thread-14856::WARNING::2013-03-18 01:54:08,513::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:54:08,515::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:54:11,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:54:11,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000985674 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:54:13,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:54:13,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00782993 s, 523 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:54:18,517::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 01:54:18,518::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:54:18,519::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:54:18,958::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:54:18,966::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:54:18,968::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:54:21,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:54:21,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125831 s, 326 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:54:23,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:54:23,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142467 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:54:31,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:54:31,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152368 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:54:33,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:54:33,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182106 s, 225 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:54:41,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:54:41,658::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858343 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:54:43,822::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 01:54:43,823::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 01:54:43,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:54:43,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000771904 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:54:43,849::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000771904 s, 2.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 01:54:43,850::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 01:54:43,850::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 01:54:43,857::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 01:54:43,858::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:54:43,859::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:54:43,859::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:54:43,859::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:54:43,860::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:54:43,861::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:54:44,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:54:44,309::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:54:44,311::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 01:54:44,313::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 01:54:44,314::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 01:54:44,315::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 01:54:44,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:54:44,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:54:44,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:54:44,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00920006 s, 445 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:54:51,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:54:51,715::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139707 s, 293 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:54:54,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:54:54,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130932 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:55:01,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:55:01,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000937675 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:55:04,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:55:04,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133357 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:55:11,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:55:11,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133901 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:55:14,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:55:14,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132692 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14857::WARNING::2013-03-18 01:55:18,994::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:55:18,999::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:55:21,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:55:21,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807411 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:55:25,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:55:25,040::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778936 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:55:29,001::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:55:29,002::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:55:29,496::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:55:29,502::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:55:29,504::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:55:31,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:55:31,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00354874 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:55:35,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:55:35,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100708 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:55:41,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:55:41,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139917 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:55:45,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:55:45,188::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151745 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:55:51,909::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 01:55:51,911::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 01:55:51,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:55:51,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00133636 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:55:51,954::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00133636 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 01:55:51,955::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 01:55:51,955::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 01:55:51,956::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 01:55:51,957::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:55:51,958::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:55:51,959::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:55:51,959::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:55:51,959::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 01:55:51,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:55:52,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:55:52,409::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:55:52,410::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 01:55:52,413::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 01:55:52,414::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 01:55:52,414::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 01:55:52,416::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:55:52,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:55:52,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:55:52,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166485 s, 246 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:55:55,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:55:55,287::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0263511 s, 155 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:56:02,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:56:02,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183343 s, 223 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:56:05,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:56:05,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786739 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:56:12,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:56:12,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000983981 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:56:15,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:56:15,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074587 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:56:22,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:56:22,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155271 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:56:25,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:56:25,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015079 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14858::WARNING::2013-03-18 01:56:29,531::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:56:29,535::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:56:32,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:56:33,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828967 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:56:35,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:56:35,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151254 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:56:39,537::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:56:39,538::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:56:40,068::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:56:40,075::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:56:40,076::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:56:43,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:56:43,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140282 s, 292 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:56:45,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:56:45,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886013 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:56:53,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:56:53,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835554 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:56:55,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:56:55,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133523 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:57:03,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:57:03,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00421763 s, 971 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:57:05,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:57:05,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885322 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:57:13,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:57:13,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861166 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:57:15,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:57:15,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079316 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:57:23,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:57:23,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00558488 s, 733 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:57:25,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:57:25,936::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132823 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:57:33,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:57:33,311::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000678979 s, 6.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:57:35,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:57:36,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176953 s, 2.3 MB/s\n'; <rc> = 0
>Thread-14859::WARNING::2013-03-18 01:57:40,101::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:57:40,105::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:57:43,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:57:43,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173202 s, 236 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:57:46,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:57:46,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795961 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:57:50,112::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:57:50,113::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:57:50,569::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:57:50,576::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:57:50,577::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:57:53,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:57:53,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142288 s, 288 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:57:56,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:57:56,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00624695 s, 656 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:58:03,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:58:03,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00664577 s, 616 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:58:06,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:58:06,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133788 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:58:13,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:58:13,511::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119285 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:58:16,281::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:58:16,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160757 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:58:23,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:58:23,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00976931 s, 419 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:58:26,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:58:26,408::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017318 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:58:33,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:58:33,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153696 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:58:36,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:58:36,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138976 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:58:43,614::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:58:43,662::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00766819 s, 534 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:58:46,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:58:46,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00212439 s, 1.9 MB/s\n'; <rc> = 0
>Thread-14860::WARNING::2013-03-18 01:58:50,608::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 01:58:50,614::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 01:58:53,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:58:53,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106356 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:58:56,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:58:56,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100544 s, 4.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 01:59:00,616::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 01:59:00,617::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 01:59:01,060::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 01:59:01,066::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 01:59:01,067::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 01:59:03,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:59:03,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00855112 s, 479 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:59:06,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:59:06,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087251 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:59:13,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:59:13,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00611715 s, 670 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:59:16,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:59:16,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800085 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:59:23,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:59:23,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787484 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:59:26,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:59:26,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000995294 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:59:33,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:59:33,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139115 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:59:36,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:59:36,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123346 s, 332 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:59:43,922::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:59:43,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00286534 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:59:46,991::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 01:59:46,993::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 01:59:46,993::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:59:47,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000786972 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:59:47,023::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000786972 s, 2.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 01:59:47,024::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 01:59:47,024::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 01:59:47,025::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 01:59:47,026::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:59:47,027::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:59:47,028::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:59:47,028::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:59:47,028::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 01:59:47,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:59:47,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:59:47,470::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 01:59:47,471::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 01:59:47,474::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 01:59:47,475::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 01:59:47,476::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 01:59:47,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:59:47,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:59:47,861::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:59:47,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184238 s, 222 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 01:59:53,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 01:59:54,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179117 s, 229 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 01:59:57,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 01:59:57,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138329 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14861::WARNING::2013-03-18 02:00:01,094::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:00:01,098::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:00:04,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:00:04,081::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00940946 s, 435 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:00:08,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:00:08,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836203 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:00:11,099::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 02:00:11,101::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:00:11,102::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:00:11,537::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:00:11,544::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:00:11,545::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:00:14,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:00:14,140::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125893 s, 325 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:00:18,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:00:18,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839303 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:00:24,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:00:24,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148023 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:00:28,209::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:00:28,236::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855955 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:00:34,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:00:34,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139233 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:00:38,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:00:38,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153612 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:00:44,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:00:44,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170592 s, 240 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:00:48,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:00:48,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000710842 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:00:54,325::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 02:00:54,327::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 02:00:54,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:00:54,366::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00120488 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:00:54,367::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00120488 s, 1.7 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 02:00:54,368::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 02:00:54,368::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 02:00:54,370::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 02:00:54,371::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:00:54,372::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:00:54,373::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:00:54,373::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:00:54,373::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:00:54,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:00:54,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:00:54,825::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:00:54,826::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 02:00:54,829::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 02:00:54,830::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 02:00:54,830::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 02:00:54,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:00:55,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:00:55,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:00:55,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0406711 s, 101 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:00:58,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:00:58,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879706 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:01:05,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:01:05,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855066 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:01:08,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:01:08,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131143 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14862::WARNING::2013-03-18 02:01:11,571::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:01:11,575::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:01:15,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:01:15,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775075 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:01:18,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:01:18,607::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875678 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:01:21,577::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:01:21,578::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:01:22,027::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:01:22,034::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:01:22,036::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:01:25,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:01:25,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159625 s, 257 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:01:28,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:01:28,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741854 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:01:35,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:01:35,450::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128838 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:01:38,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:01:38,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160135 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:01:45,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:01:45,484::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877823 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:01:48,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:01:48,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137944 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:01:55,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:01:55,529::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124366 s, 329 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:01:58,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:01:58,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089258 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:02:05,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:02:05,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746517 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:02:08,944::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:02:08,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133763 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:02:15,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:02:15,607::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131824 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:02:19,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:02:19,060::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134872 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14863::WARNING::2013-03-18 02:02:22,062::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:02:22,066::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:02:25,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:02:25,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000955198 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:02:29,108::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:02:29,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000959706 s, 4.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:02:32,068::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:02:32,069::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:02:32,568::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:02:32,574::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:02:32,575::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:02:35,647::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:02:35,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129974 s, 315 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:02:39,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:02:39,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000952416 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:02:45,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:02:45,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000927824 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:02:49,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:02:49,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743562 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:02:55,732::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:02:55,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101445 s, 404 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:02:59,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:02:59,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835135 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:03:05,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:03:05,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000710016 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:03:09,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:03:09,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105699 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:03:15,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:03:15,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135231 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:03:19,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:03:19,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011986 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:03:25,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:03:25,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127369 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:03:29,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:03:29,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122255 s, 3.4 MB/s\n'; <rc> = 0
>Thread-14864::WARNING::2013-03-18 02:03:32,611::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:03:32,615::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:03:35,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:03:35,963::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0272066 s, 151 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:03:39,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:03:39,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092968 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:03:42,616::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:03:42,618::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:03:43,089::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:03:43,095::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:03:43,097::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:03:45,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:03:46,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017297 s, 237 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:03:49,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:03:49,742::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160231 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:03:56,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:03:56,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169236 s, 242 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:03:59,782::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:03:59,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00885338 s, 463 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:04:06,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:04:06,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828029 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:04:09,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:04:09,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000991703 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:04:16,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:04:16,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164285 s, 249 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:04:19,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:04:20,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177071 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:04:26,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:04:26,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139833 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:04:30,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:04:30,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160786 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:04:36,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:04:36,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180078 s, 227 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:04:40,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:04:40,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150776 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14865::WARNING::2013-03-18 02:04:43,122::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:04:43,126::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:04:46,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:04:46,328::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901304 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:04:50,229::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 02:04:50,230::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 02:04:50,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:04:50,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000864371 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:04:50,257::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000864371 s, 2.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 02:04:50,258::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 02:04:50,258::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 02:04:50,260::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 02:04:50,262::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:04:50,263::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:04:50,263::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:04:50,263::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:04:50,264::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:04:50,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:04:50,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:04:50,701::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:04:50,702::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 02:04:50,705::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 02:04:50,706::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 02:04:50,706::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 02:04:50,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:04:51,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:04:51,070::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:04:51,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00765777 s, 535 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:04:53,128::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:04:53,129::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:04:53,569::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:04:53,575::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:04:53,576::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:04:56,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:04:56,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0267962 s, 153 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:05:01,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:05:01,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125269 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:05:06,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:05:06,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805027 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:05:11,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:05:11,270::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012443 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:05:16,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:05:16,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147598 s, 278 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:05:21,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:05:21,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901133 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:05:26,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:05:26,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142525 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:05:31,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:05:31,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00455609 s, 899 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:05:36,528::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:05:36,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161649 s, 253 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:05:41,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:05:41,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013372 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:05:46,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:05:46,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116212 s, 352 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:05:51,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:05:51,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759778 s, 5.4 MB/s\n'; <rc> = 0
>Thread-14866::WARNING::2013-03-18 02:05:53,601::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:05:53,604::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:05:56,633::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 02:05:56,634::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 02:05:56,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:05:56,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00088401 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:05:56,665::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00088401 s, 2.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 02:05:56,665::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 02:05:56,666::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 02:05:56,668::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 02:05:56,669::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:05:56,670::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:05:56,671::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:05:56,671::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:05:56,672::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:05:56,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:05:57,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:05:57,103::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:05:57,104::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 02:05:57,106::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 02:05:57,107::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 02:05:57,108::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 02:05:57,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:05:57,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:05:57,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:05:57,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114008 s, 359 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:06:01,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:06:01,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141734 s, 2.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:06:03,606::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 02:06:03,607::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:06:03,608::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:06:04,084::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:06:04,091::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:06:04,092::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:06:07,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:06:07,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113784 s, 360 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:06:11,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:06:11,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017121 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:06:17,560::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:06:17,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814027 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:06:21,786::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:06:21,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139664 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:06:27,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:06:27,636::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138863 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:06:31,871::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:06:31,899::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896332 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:06:37,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:06:37,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768679 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:06:41,938::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:06:41,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147181 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:06:47,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:06:47,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0027053 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:06:52,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:06:52,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138934 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:06:57,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:06:57,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010093 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:07:02,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:07:02,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100035 s, 4.1 MB/s\n'; <rc> = 0
>Thread-14867::WARNING::2013-03-18 02:07:04,129::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:07:04,134::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:07:07,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:07:07,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792966 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:07:12,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:07:12,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113664 s, 3.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:07:14,140::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:07:14,142::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:07:14,586::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:07:14,593::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:07:14,594::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:07:17,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:07:17,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015079 s, 272 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:07:22,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:07:22,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830478 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:07:27,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:07:27,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828733 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:07:32,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:07:32,366::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820743 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:07:37,875::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:07:37,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139744 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:07:42,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:07:42,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013958 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:07:47,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:07:47,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856117 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:07:52,490::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:07:52,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835342 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:07:57,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:07:57,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104329 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:08:02,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:08:02,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843751 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:08:07,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:08:08,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823302 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:08:12,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:08:12,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000939778 s, 4.4 MB/s\n'; <rc> = 0
>Thread-14868::WARNING::2013-03-18 02:08:14,621::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:08:14,624::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:08:18,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:08:18,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853986 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:08:22,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:08:22,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00073669 s, 5.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:08:24,628::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:08:24,629::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:08:25,172::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:08:25,179::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:08:25,180::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:08:28,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:08:28,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0255355 s, 160 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:08:32,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:08:32,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861542 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:08:38,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:08:38,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140408 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:08:42,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:08:42,936::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852597 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:08:48,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:08:48,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894189 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:08:52,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:08:53,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833571 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:08:58,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:08:58,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128465 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:09:03,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:09:03,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834527 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:09:08,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:09:08,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139272 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:09:13,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:09:13,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130438 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:09:18,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:09:18,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810907 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:09:23,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:09:23,236::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155119 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14869::WARNING::2013-03-18 02:09:25,212::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:09:25,216::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:09:28,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:09:28,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834422 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:09:33,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:09:33,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853808 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:09:35,218::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:09:35,219::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:09:35,637::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:09:35,643::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:09:35,645::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:09:38,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:09:38,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150623 s, 272 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:09:43,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:09:43,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859051 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:09:48,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:09:48,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081367 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:09:53,411::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 02:09:53,413::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 02:09:53,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:09:53,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000699236 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:09:53,441::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000699236 s, 2.9 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 02:09:53,441::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 02:09:53,442::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 02:09:53,443::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 02:09:53,444::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:09:53,445::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:09:53,446::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:09:53,446::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:09:53,446::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:09:53,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:09:53,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:09:53,876::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:09:53,877::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 02:09:53,880::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 02:09:53,881::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 02:09:53,881::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 02:09:53,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:09:54,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:09:54,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:09:54,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00911295 s, 449 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:09:58,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:09:58,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00905847 s, 452 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:10:04,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:10:04,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143226 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:10:08,524::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:10:08,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130542 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:10:14,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:10:14,448::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853214 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:10:18,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:10:18,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832323 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:10:24,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:10:24,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144018 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:10:28,612::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:10:28,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132478 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:10:34,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:10:34,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00202216 s, 2.0 MB/s\n'; <rc> = 0
>Thread-14870::WARNING::2013-03-18 02:10:35,675::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:10:35,680::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:10:38,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:10:38,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786711 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:10:44,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:10:44,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878057 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:10:45,682::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:10:45,684::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:10:46,109::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:10:46,115::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:10:46,117::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:10:48,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:10:48,742::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150538 s, 272 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:10:54,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:10:54,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000732667 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:10:58,747::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 02:10:58,749::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 02:10:58,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:10:58,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00117616 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:10:58,783::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00117616 s, 1.7 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 02:10:58,784::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 02:10:58,784::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 02:10:58,786::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 02:10:58,787::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:10:58,787::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:10:58,788::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:10:58,788::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:10:58,789::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:10:58,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:10:59,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:10:59,218::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:10:59,219::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 02:10:59,222::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 02:10:59,223::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 02:10:59,223::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 02:10:59,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:10:59,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:10:59,605::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:10:59,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192358 s, 213 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:11:05,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:11:05,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0278368 s, 147 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:11:09,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:11:09,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156798 s, 261 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:11:15,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:11:15,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091533 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:11:19,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:11:19,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138529 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:11:25,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:11:25,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145308 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:11:29,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:11:29,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176841 s, 232 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:11:35,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:11:35,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000707823 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:11:39,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:11:39,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00206396 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:11:45,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:11:45,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00510927 s, 802 kB/s\n'; <rc> = 0
>Thread-14871::WARNING::2013-03-18 02:11:46,153::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:11:46,157::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:11:49,875::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:11:49,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177037 s, 231 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:11:55,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:11:55,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848338 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:11:56,159::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 02:11:56,160::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:11:56,161::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:11:56,648::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:11:56,654::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:11:56,655::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:11:59,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:11:59,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189748 s, 216 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:12:05,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:12:06,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848069 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:12:09,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:12:10,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160094 s, 256 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:12:16,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:12:16,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122507 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:12:20,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:12:20,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137927 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:12:26,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:12:26,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806969 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:12:30,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:12:30,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174348 s, 235 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:12:36,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:12:36,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870512 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:12:40,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:12:40,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014338 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:12:46,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:12:46,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121577 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:12:50,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:12:50,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149723 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:12:56,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:12:56,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795288 s, 5.2 MB/s\n'; <rc> = 0
>Thread-14872::WARNING::2013-03-18 02:12:56,683::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:12:56,686::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:13:00,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:13:00,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087802 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:13:06,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:13:06,711::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:13:06,735::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:13:06,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131206 s, 312 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:13:07,232::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:13:07,238::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:13:07,239::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:13:10,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:13:10,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199571 s, 205 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:13:16,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:13:16,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00338227 s, 1.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:13:20,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:13:20,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826912 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:13:26,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:13:26,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777855 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:13:30,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:13:30,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080883 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:13:36,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:13:37,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00293497 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:13:40,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:13:40,481::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143414 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:13:47,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:13:47,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00073859 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:13:50,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:13:50,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082233 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:13:57,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:13:57,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016285 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:14:00,524::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:14:00,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160892 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:14:07,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:14:07,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00180763 s, 2.3 MB/s\n'; <rc> = 0
>Thread-14873::WARNING::2013-03-18 02:14:07,273::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:14:07,286::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:14:10,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:14:10,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103616 s, 4.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:14:17,290::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:14:17,291::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:14:17,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:14:17,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162472 s, 2.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:14:17,706::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:14:17,711::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:14:17,713::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:14:20,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:14:20,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138312 s, 296 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:14:27,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:14:27,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013853 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:14:30,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:14:30,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00823736 s, 497 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:14:37,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:14:37,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000935103 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:14:40,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:14:40,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113475 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:14:47,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:14:47,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130225 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:14:50,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:14:50,843::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140335 s, 292 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:14:57,686::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 02:14:57,688::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 02:14:57,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:14:57,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000897207 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:14:57,721::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000897207 s, 2.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 02:14:57,722::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 02:14:57,722::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 02:14:57,724::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 02:14:57,724::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:14:57,725::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:14:57,726::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:14:57,726::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:14:57,726::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:14:57,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:14:58,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:14:58,150::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:14:58,151::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 02:14:58,153::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 02:14:58,154::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 02:14:58,155::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 02:14:58,156::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:14:58,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:14:58,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:14:58,593::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114589 s, 357 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:15:00,859::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:15:00,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177474 s, 231 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:15:08,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:15:08,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831422 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:15:10,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:15:10,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148262 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14874::WARNING::2013-03-18 02:15:17,742::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:15:17,746::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 02:15:18,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:15:18,734::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819816 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:15:20,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:15:21,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144072 s, 2.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:15:27,748::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:15:27,749::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:15:28,191::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:15:28,198::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:15:28,199::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:15:28,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:15:28,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0227527 s, 180 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:15:31,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:15:31,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113072 s, 362 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:15:38,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:15:38,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00200897 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:15:41,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:15:41,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798494 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:15:48,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:15:48,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000922983 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:15:51,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:15:51,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128315 s, 319 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:15:59,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:15:59,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820304 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:16:01,184::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 02:16:01,186::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 02:16:01,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:16:01,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00133728 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:16:01,232::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00133728 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 02:16:01,233::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 02:16:01,233::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 02:16:01,234::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 02:16:01,235::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:16:01,236::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:16:01,236::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:16:01,237::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:16:01,237::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:16:01,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:16:01,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:16:01,862::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:16:01,863::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 02:16:01,866::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 02:16:01,867::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 02:16:01,867::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 02:16:01,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:16:02,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:16:02,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:16:02,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0046469 s, 881 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:16:09,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:16:09,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160544 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:16:12,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:16:12,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135365 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:16:19,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:16:19,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776659 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:16:22,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:16:22,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154458 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14875::WARNING::2013-03-18 02:16:28,226::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:16:28,230::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 02:16:29,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:16:29,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777953 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:16:32,395::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:16:32,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782038 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:16:38,232::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:16:38,233::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:16:38,640::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:16:38,647::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:16:38,648::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:16:39,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:16:39,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138701 s, 295 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:16:42,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:16:42,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014813 s, 277 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:16:49,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:16:49,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162888 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:16:52,481::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:16:52,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806704 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:16:59,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:16:59,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900852 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:17:02,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:17:02,548::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863971 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:17:09,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:17:09,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00294112 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:17:12,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:17:12,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156487 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:17:19,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:17:19,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110233 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:17:22,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:17:22,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821912 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:17:29,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:17:29,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137762 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:17:32,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:17:32,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000961461 s, 4.3 MB/s\n'; <rc> = 0
>Thread-14876::WARNING::2013-03-18 02:17:38,675::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:17:38,680::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 02:17:39,797::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:17:39,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000742877 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:17:42,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:17:42,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892576 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:17:48,682::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 02:17:48,683::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:17:48,684::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:17:49,107::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:17:49,114::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:17:49,115::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:17:49,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:17:49,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123659 s, 331 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:17:52,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:17:52,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00749256 s, 547 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:17:59,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:17:59,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757748 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:18:02,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:18:02,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139481 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:18:10,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:18:10,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013131 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:18:12,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:18:12,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011023 s, 372 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:18:20,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:18:20,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142239 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:18:22,875::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:18:22,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904009 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:18:30,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:18:30,223::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891923 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:18:32,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:18:32,944::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141341 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:18:40,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:18:40,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132572 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:18:42,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:18:42,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164393 s, 249 kB/s\n'; <rc> = 0
>Thread-14877::WARNING::2013-03-18 02:18:49,141::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:18:49,145::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 02:18:50,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:18:50,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178735 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:18:53,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:18:53,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833396 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:18:59,152::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:18:59,153::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:18:59,582::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:18:59,589::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:18:59,590::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:19:00,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:19:00,478::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01374 s, 298 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:19:03,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:19:03,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154987 s, 264 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:19:10,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:19:10,562::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134209 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:19:13,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:19:13,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805965 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:19:20,603::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:19:20,633::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810379 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:19:23,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:19:23,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808926 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:19:30,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:19:30,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840974 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:19:33,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:19:33,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114041 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:19:40,742::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:19:40,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102466 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:19:43,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:19:43,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188179 s, 218 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:19:50,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:19:50,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143167 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:19:53,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:19:53,333::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00293907 s, 1.4 MB/s\n'; <rc> = 0
>Thread-14878::WARNING::2013-03-18 02:19:59,619::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:19:59,624::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 02:20:00,888::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 02:20:00,890::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 02:20:00,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:20:00,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000800208 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:20:00,917::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000800208 s, 2.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 02:20:00,917::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 02:20:00,917::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 02:20:00,919::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 02:20:00,920::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:20:00,921::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:20:00,921::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:20:00,921::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:20:00,922::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:20:00,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:20:01,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:20:01,373::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:20:01,374::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 02:20:01,377::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 02:20:01,378::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 02:20:01,379::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 02:20:01,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:20:01,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:20:01,814::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:20:01,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0096274 s, 425 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:20:03,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:20:03,382::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148658 s, 276 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:20:09,626::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:20:09,627::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:20:10,064::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:20:10,070::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:20:10,072::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:20:11,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:20:11,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110677 s, 370 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:20:13,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:20:13,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145946 s, 281 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:20:21,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:20:22,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145169 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:20:23,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:20:23,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887408 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:20:32,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:20:32,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156529 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:20:33,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:20:33,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104177 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:20:42,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:20:42,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082943 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:20:43,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:20:43,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863382 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:20:52,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:20:52,260::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134192 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:20:53,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:20:53,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137072 s, 299 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:21:02,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:21:02,333::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101562 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:21:03,596::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 02:21:03,598::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 02:21:03,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:21:03,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0013757 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:21:03,641::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0013757 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 02:21:03,641::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 02:21:03,641::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 02:21:03,643::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 02:21:03,644::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:21:03,645::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:21:03,645::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:21:03,645::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:21:03,646::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:21:03,647::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:21:04,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:21:04,112::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:21:04,113::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 02:21:04,115::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 02:21:04,117::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 02:21:04,117::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 02:21:04,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:21:04,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:21:04,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:21:04,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173037 s, 237 kB/s\n'; <rc> = 0
>Thread-14879::WARNING::2013-03-18 02:21:10,100::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:21:10,105::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 02:21:12,373::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:21:12,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830791 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:21:14,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:21:14,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000957968 s, 4.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:21:20,114::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:21:20,115::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:21:20,526::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:21:20,533::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:21:20,534::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:21:22,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:21:22,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0221179 s, 185 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:21:24,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:21:24,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167457 s, 245 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:21:32,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:21:32,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106411 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:21:34,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:21:34,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136261 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:21:42,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:21:42,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886939 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:21:44,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:21:44,818::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123985 s, 330 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:21:52,681::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:21:52,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158312 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:21:54,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:21:54,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143947 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:22:02,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:22:02,777::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799162 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:22:04,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:22:04,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014902 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:22:12,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:22:12,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146421 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:22:14,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:22:14,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157128 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14880::WARNING::2013-03-18 02:22:20,560::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:22:20,563::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 02:22:22,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:22:22,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904774 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:22:24,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:22:24,997::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888301 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:22:30,565::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:22:30,566::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:22:31,030::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:22:31,037::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:22:31,038::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:22:32,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:22:33,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155924 s, 263 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:22:35,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:22:35,054::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134917 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:22:43,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:22:43,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865349 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:22:45,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:22:45,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151515 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:22:53,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:22:53,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145251 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:22:55,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:22:55,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186452 s, 220 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:23:03,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:23:03,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146452 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:23:05,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:23:05,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076942 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:23:13,281::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:23:13,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855196 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:23:15,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:23:15,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145797 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:23:23,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:23:23,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882257 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:23:25,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:23:25,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167371 s, 245 kB/s\n'; <rc> = 0
>Thread-14881::WARNING::2013-03-18 02:23:31,071::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:23:31,082::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 02:23:33,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:23:33,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834005 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:23:35,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:23:35,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885792 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:23:41,085::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 02:23:41,086::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:23:41,087::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:23:41,589::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:23:41,596::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:23:41,598::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:23:43,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:23:43,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100534 s, 407 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:23:45,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:23:45,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:23:53,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:23:53,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788305 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:23:55,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:23:55,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808019 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:24:03,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:24:03,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136482 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:24:05,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:24:05,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082724 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:24:13,732::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:24:13,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896006 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:24:15,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:24:15,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143329 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:24:23,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:24:23,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013313 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:24:25,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:24:25,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160487 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:24:33,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:24:33,928::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015379 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:24:35,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:24:35,602::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128217 s, 3.2 MB/s\n'; <rc> = 0
>Thread-14882::WARNING::2013-03-18 02:24:41,629::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:24:41,632::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 02:24:43,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:24:44,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139577 s, 293 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:24:45,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:24:45,657::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818432 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:24:51,635::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:24:51,636::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:24:52,092::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:24:52,100::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:24:52,102::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:24:54,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:24:54,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127174 s, 322 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:24:55,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:24:55,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013068 s, 313 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:25:04,173::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 02:25:04,175::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 02:25:04,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:25:04,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0103704 s, 197 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:25:04,212::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0103704 s, 197 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 02:25:04,212::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 02:25:04,213::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 02:25:04,214::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 02:25:04,215::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:25:04,216::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:25:04,217::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:25:04,217::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:25:04,217::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:25:04,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:25:04,657::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:25:04,661::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:25:04,662::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 02:25:04,664::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 02:25:04,665::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 02:25:04,666::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 02:25:04,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:25:05,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:25:05,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:25:05,157::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0434061 s, 94.4 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:25:05,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:25:05,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117951 s, 347 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:25:15,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:25:15,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091576 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:25:15,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:25:15,820::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858451 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:25:25,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:25:25,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149318 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:25:25,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:25:25,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156701 s, 261 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:25:35,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:25:35,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749676 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:25:35,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:25:35,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873002 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:25:45,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:25:45,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000949735 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:25:45,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:25:45,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100194 s, 409 kB/s\n'; <rc> = 0
>Thread-14883::WARNING::2013-03-18 02:25:52,150::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:25:52,231::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 02:25:55,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:25:55,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916795 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:25:55,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:25:56,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103584 s, 395 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:26:02,235::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:26:02,242::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:26:02,757::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:26:02,766::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:26:02,767::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:26:05,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:26:05,633::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127598 s, 321 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:26:06,022::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 02:26:06,023::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 02:26:06,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:26:06,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0155348 s, 132 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:26:06,065::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0155348 s, 132 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 02:26:06,065::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 02:26:06,066::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 02:26:06,068::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 02:26:06,069::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:26:06,070::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:26:06,070::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:26:06,071::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:26:06,071::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:26:06,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:26:06,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:26:06,501::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:26:06,502::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 02:26:06,505::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 02:26:06,506::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 02:26:06,506::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 02:26:06,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:26:06,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:26:06,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:26:06,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00193509 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:26:15,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:26:15,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080592 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:26:16,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:26:16,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109561 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:26:25,740::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:26:25,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828999 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:26:26,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:26:26,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866217 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:26:35,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:26:35,837::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768045 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:26:37,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:26:37,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879684 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:26:45,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:26:45,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000699854 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:26:47,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:26:47,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0201365 s, 203 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:26:55,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:26:56,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147232 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:26:57,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:26:57,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000736336 s, 5.6 MB/s\n'; <rc> = 0
>Thread-14884::WARNING::2013-03-18 02:27:02,794::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:27:02,800::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 02:27:06,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:27:06,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806335 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:27:07,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:27:07,175::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886808 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:27:12,803::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:27:12,804::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:27:13,322::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:27:13,328::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:27:13,329::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:27:16,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:27:16,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0174777 s, 234 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:27:17,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:27:17,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824166 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:27:26,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:27:26,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865992 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:27:27,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:27:27,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873754 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:27:36,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:27:36,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798385 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:27:37,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:27:37,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077832 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:27:46,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:27:46,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782158 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:27:47,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:27:47,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148974 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:27:56,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:27:56,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158822 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:27:57,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:27:57,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013469 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:28:06,490::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:28:06,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133389 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:28:07,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:28:07,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000967959 s, 4.2 MB/s\n'; <rc> = 0
>Thread-14885::WARNING::2013-03-18 02:28:13,357::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:28:13,364::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 02:28:16,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:28:16,598::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775518 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:28:17,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:28:17,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830197 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:28:23,366::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:28:23,368::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:28:23,783::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:28:23,790::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:28:23,791::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:28:26,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:28:26,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00958927 s, 427 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:28:27,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:28:27,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811029 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:28:36,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:28:36,756::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135215 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:28:37,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:28:37,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753829 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:28:46,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:28:46,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000979741 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:28:47,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:28:47,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157872 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:28:56,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:28:56,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014341 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:28:57,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:28:57,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782633 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:29:06,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:29:06,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770288 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:29:07,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:29:07,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132279 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:29:17,019::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:29:17,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014269 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:29:17,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:29:17,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151303 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14886::WARNING::2013-03-18 02:29:23,820::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:29:23,824::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 02:29:27,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:29:27,125::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000978002 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:29:27,732::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:29:27,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000930138 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:29:33,825::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 02:29:33,827::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:29:33,828::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:29:34,297::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:29:34,305::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:29:34,306::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:29:37,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:29:37,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078189 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:29:37,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:29:37,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157654 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:29:47,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:29:47,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106096 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:29:47,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:29:47,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00098214 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:29:57,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:29:57,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846486 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:29:57,922::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:29:57,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156024 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:30:07,372::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 02:30:07,374::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 02:30:07,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:30:07,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00891645 s, 230 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:30:07,428::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00891645 s, 230 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 02:30:07,429::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 02:30:07,429::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 02:30:07,430::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 02:30:07,431::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:30:07,432::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:30:07,432::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:30:07,433::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:30:07,433::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:30:07,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:30:07,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:30:07,828::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:30:07,829::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 02:30:07,831::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 02:30:07,833::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 02:30:07,833::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 02:30:07,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:30:07,978::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:30:08,040::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156613 s, 262 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:30:08,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:30:08,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:30:08,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148775 s, 275 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:30:18,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:30:18,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171862 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:30:18,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:30:18,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141864 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:30:28,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:30:28,123::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085549 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:30:28,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:30:28,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105733 s, 3.9 MB/s\n'; <rc> = 0
>Thread-14887::WARNING::2013-03-18 02:30:34,334::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:30:34,339::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:30:38,129::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:30:38,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902507 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:30:38,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:30:38,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148186 s, 2.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:30:44,342::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:30:44,343::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:30:44,792::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:30:44,800::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:30:44,802::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:30:48,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:30:48,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0195281 s, 210 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:30:48,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:30:48,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771886 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:30:58,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:30:58,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0032762 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:30:58,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:30:58,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102825 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:31:08,256::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 02:31:08,257::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 02:31:08,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:31:08,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00152288 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:31:08,304::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00152288 s, 1.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 02:31:08,305::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 02:31:08,305::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 02:31:08,307::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 02:31:08,308::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:31:08,308::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:31:08,309::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:31:08,309::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:31:08,309::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:31:08,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:31:08,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:31:08,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:31:08,740::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:31:08,741::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-20::DEBUG::2013-03-18 02:31:08,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176607 s, 232 kB/s\n'; <rc> = 0
>Thread-22::WARNING::2013-03-18 02:31:08,750::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 02:31:08,753::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 02:31:08,755::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 02:31:08,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:31:09,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:31:09,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:31:09,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159189 s, 257 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:31:18,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:31:18,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000731037 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:31:19,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:31:19,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139753 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:31:28,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:31:28,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00316224 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:31:29,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:31:29,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130946 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:31:39,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:31:39,060::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160986 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:31:39,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:31:39,399::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134288 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14888::WARNING::2013-03-18 02:31:44,830::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:31:44,835::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 02:31:49,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:31:49,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000958182 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:31:49,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:31:49,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00243551 s, 1.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:31:54,837::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:31:54,838::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:31:55,252::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:31:55,258::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:31:55,260::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:31:59,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:31:59,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139376 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:31:59,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:31:59,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157082 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:32:09,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:32:09,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153541 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:32:09,490::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:32:09,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135619 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:32:19,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:32:19,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790987 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:32:19,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:32:19,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00742403 s, 552 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:32:29,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:32:29,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155544 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:32:29,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:32:29,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00294137 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:32:39,493::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:32:39,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145418 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:32:39,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:32:39,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860634 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:32:49,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:32:49,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771053 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:32:49,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:32:49,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130064 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14889::WARNING::2013-03-18 02:32:55,288::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:32:55,294::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 02:32:59,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:32:59,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799151 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:32:59,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:32:59,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751074 s, 5.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:33:05,297::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:33:05,298::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:33:05,695::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:33:05,701::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:33:05,703::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:33:09,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:33:09,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:33:09,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116898 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:33:09,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00684282 s, 599 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:33:19,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:33:19,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:33:19,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0213952 s, 191 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:33:19,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122221 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:33:29,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:33:29,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:33:29,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130072 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:33:30,010::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842214 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:33:40,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:33:40,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:33:40,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013971 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:33:40,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114377 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:33:50,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:33:50,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:33:50,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131135 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:33:50,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108916 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:34:00,271::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:34:00,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:34:00,364::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089715 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:34:00,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112631 s, 3.6 MB/s\n'; <rc> = 0
>Thread-14890::WARNING::2013-03-18 02:34:05,730::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:34:05,734::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:34:10,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:34:10,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128936 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:34:10,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:34:10,448::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000888834 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:34:15,737::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:34:15,738::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:34:16,253::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:34:16,260::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:34:16,261::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:34:20,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:34:20,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140887 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:34:20,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:34:20,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143351 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:34:30,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:34:30,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850139 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:34:30,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:34:30,607::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838574 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:34:40,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:34:40,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168485 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:34:40,647::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:34:40,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016007 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:34:50,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:34:50,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011792 s, 347 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:34:50,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:34:50,786::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103437 s, 396 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:35:00,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:35:00,700::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116827 s, 351 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:35:00,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:35:00,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802045 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:35:10,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:35:10,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000666208 s, 6.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:35:10,905::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 02:35:10,907::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 02:35:10,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:35:10,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00132101 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:35:10,950::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00132101 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 02:35:10,950::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 02:35:10,950::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 02:35:10,952::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 02:35:10,953::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:35:10,954::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:35:10,954::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:35:10,955::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:35:10,955::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:35:10,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:35:11,392::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:35:11,400::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:35:11,401::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 02:35:11,403::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 02:35:11,405::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 02:35:11,406::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 02:35:11,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:35:11,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:35:11,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:35:11,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00395774 s, 1.0 MB/s\n'; <rc> = 0
>Thread-14891::WARNING::2013-03-18 02:35:16,287::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:35:16,290::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:35:20,781::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:35:20,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0447588 s, 91.5 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:35:21,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:35:21,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811219 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:35:26,292::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 02:35:26,293::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:35:26,294::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:35:26,737::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:35:26,743::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:35:26,745::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:35:30,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:35:30,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0070063 s, 585 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:35:31,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:35:31,928::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886812 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:35:40,934::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:35:40,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151807 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:35:41,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:35:41,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116285 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:35:51,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:35:51,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000703426 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:35:52,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:35:52,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000938451 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:36:01,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:36:01,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000955456 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:36:02,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:36:02,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00227037 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:36:11,096::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 02:36:11,097::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 02:36:11,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:36:11,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000822284 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:36:11,131::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000822284 s, 2.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 02:36:11,131::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 02:36:11,131::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 02:36:11,133::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 02:36:11,134::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:36:11,135::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:36:11,135::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:36:11,136::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:36:11,136::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:36:11,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:36:11,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:36:11,582::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:36:11,583::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 02:36:11,585::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 02:36:11,586::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 02:36:11,587::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 02:36:11,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:36:11,950::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:36:11,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:36:11,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00714578 s, 573 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:36:12,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:36:12,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00765276 s, 535 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:36:21,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:36:22,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868536 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:36:22,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:36:22,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174363 s, 2.3 MB/s\n'; <rc> = 0
>Thread-14892::WARNING::2013-03-18 02:36:26,768::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:36:26,771::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:36:32,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:36:32,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132498 s, 309 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:36:32,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:36:32,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000992648 s, 4.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:36:36,773::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:36:36,775::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:36:37,203::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:36:37,209::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:36:37,211::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:36:42,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:36:42,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865975 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:36:42,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:36:42,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896662 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:36:52,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:36:52,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847242 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:36:52,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:36:52,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786152 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:37:02,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:37:02,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815925 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:37:02,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:37:02,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000930424 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:37:12,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:37:12,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890494 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:37:12,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:37:12,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144473 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:37:22,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:37:22,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015303 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:37:22,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:37:22,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747046 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:37:32,271::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:37:32,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787549 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:37:32,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:37:32,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165091 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14893::WARNING::2013-03-18 02:37:37,248::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:37:37,254::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:37:42,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:37:42,365::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173623 s, 236 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:37:42,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:37:42,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149149 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:37:47,256::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:37:47,258::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:37:47,713::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:37:47,720::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:37:47,721::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:37:52,372::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:37:52,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141861 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:37:53,021::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:37:53,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102105 s, 401 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:38:02,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:38:02,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150026 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:38:03,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:38:03,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135932 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:38:12,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:38:12,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140608 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:38:13,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:38:13,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147884 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:38:22,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:38:22,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865801 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:38:23,281::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:38:23,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142066 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:38:32,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:38:32,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136157 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:38:33,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:38:33,402::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135977 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:38:42,596::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:38:42,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015144 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:38:43,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:38:43,483::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141275 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14894::WARNING::2013-03-18 02:38:47,747::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:38:47,753::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:38:52,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:38:52,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809529 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:38:53,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:38:53,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00209115 s, 2.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:38:57,756::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:38:57,758::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:38:58,184::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:38:58,190::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:38:58,191::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:39:02,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:39:02,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014655 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:39:03,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:39:03,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932942 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:39:12,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:39:12,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084337 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:39:13,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:39:13,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00192952 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:39:22,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:39:22,819::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00090672 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:39:23,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:39:23,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00259847 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:39:32,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:39:32,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00180698 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:39:33,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:39:33,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150799 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:39:42,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:39:42,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139161 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:39:43,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:39:43,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000955233 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:39:52,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:39:52,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00361942 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:39:54,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:39:54,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.012083 s, 339 kB/s\n'; <rc> = 0
>Thread-14895::WARNING::2013-03-18 02:39:58,218::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:39:58,222::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:40:02,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:40:03,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001665 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:40:04,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:40:04,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150609 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:40:08,225::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:40:08,226::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:40:08,627::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:40:08,633::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:40:08,635::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:40:13,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:40:13,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161986 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:40:14,176::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 02:40:14,178::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 02:40:14,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:40:14,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00145059 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:40:14,225::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00145059 s, 1.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 02:40:14,226::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 02:40:14,226::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 02:40:14,228::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 02:40:14,229::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:40:14,229::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:40:14,230::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:40:14,230::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:40:14,231::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:40:14,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:40:14,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:40:14,630::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:40:14,631::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 02:40:14,634::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 02:40:14,635::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 02:40:14,635::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 02:40:14,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:40:15,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:40:15,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:40:15,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161407 s, 254 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:40:23,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:40:23,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000976297 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:40:25,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:40:25,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00070702 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:40:33,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:40:33,188::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103418 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:40:35,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:40:35,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139986 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:40:43,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:40:43,238::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00250435 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:40:45,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:40:45,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077208 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:40:53,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:40:53,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00190498 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:40:55,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:40:55,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103954 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:41:03,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:41:03,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011864 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:41:05,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:41:05,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00231323 s, 1.8 MB/s\n'; <rc> = 0
>Thread-14896::WARNING::2013-03-18 02:41:08,664::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:41:08,669::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:41:13,352::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 02:41:13,353::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 02:41:13,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:41:13,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00142044 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:41:13,389::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00142044 s, 1.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 02:41:13,390::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 02:41:13,390::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 02:41:13,392::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 02:41:13,393::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:41:13,394::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:41:13,394::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:41:13,394::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:41:13,395::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:41:13,396::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:41:13,820::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:41:13,825::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:41:13,826::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 02:41:13,828::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 02:41:13,829::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 02:41:13,830::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 02:41:13,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:41:14,191::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:41:14,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:41:14,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00898911 s, 456 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:41:15,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:41:15,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110777 s, 370 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:41:18,671::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 02:41:18,672::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:41:18,673::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:41:19,090::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:41:19,097::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:41:19,098::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:41:24,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:41:24,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00839318 s, 488 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:41:25,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:41:25,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816778 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:41:34,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:41:34,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083422 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:41:35,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:41:35,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122011 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:41:44,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:41:44,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153322 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:41:45,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:41:45,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150169 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:41:54,352::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:41:54,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899055 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:41:55,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:41:55,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145206 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:42:04,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:42:04,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001273 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:42:05,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:42:05,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138631 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:42:14,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:42:14,478::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169123 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:42:16,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:42:16,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139297 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14897::WARNING::2013-03-18 02:42:19,126::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:42:19,130::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:42:24,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:42:24,521::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00930746 s, 440 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:42:26,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:42:26,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754801 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:42:29,132::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:42:29,133::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:42:29,571::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:42:29,577::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:42:29,579::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:42:34,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:42:34,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822307 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:42:36,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:42:36,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078461 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:42:44,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:42:44,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164957 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:42:46,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:42:46,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154226 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:42:54,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:42:54,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753759 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:42:56,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:42:56,409::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168725 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:43:04,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:43:04,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838258 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:43:06,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:43:06,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149746 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:43:14,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:43:14,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014181 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:43:16,528::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:43:16,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163154 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:43:24,730::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:43:24,766::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155955 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:43:26,614::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:43:26,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00180903 s, 2.3 MB/s\n'; <rc> = 0
>Thread-14898::WARNING::2013-03-18 02:43:29,617::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:43:29,620::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:43:34,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:43:34,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783154 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:43:36,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:43:36,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000739763 s, 5.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:43:39,622::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:43:39,624::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:43:40,030::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:43:40,036::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:43:40,038::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:43:44,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:43:44,853::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148403 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:43:46,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:43:46,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000949092 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:43:54,859::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:43:54,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780732 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:43:56,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:43:56,866::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810446 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:44:04,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:44:04,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013577 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:44:06,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:44:06,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014345 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:44:14,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:44:14,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142906 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:44:16,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:44:17,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823948 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:44:24,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:44:25,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838974 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:44:27,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:44:27,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872517 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:44:35,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:44:35,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152972 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:44:37,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:44:37,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00242237 s, 1.7 MB/s\n'; <rc> = 0
>Thread-14899::WARNING::2013-03-18 02:44:40,070::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:44:40,074::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:44:45,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:44:45,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0262013 s, 156 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:44:47,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:44:47,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819086 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:44:50,076::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:44:50,078::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:44:50,503::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:44:50,510::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:44:50,512::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:44:55,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:44:55,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140079 s, 292 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:44:57,292::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:44:57,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139357 s, 294 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:45:05,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:45:05,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129243 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:45:07,372::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:45:07,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133739 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:45:15,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:45:15,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861091 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:45:17,462::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 02:45:17,463::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 02:45:17,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:45:17,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00144135 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:45:17,509::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00144135 s, 1.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 02:45:17,510::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 02:45:17,510::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 02:45:17,512::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 02:45:17,513::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:45:17,513::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:45:17,514::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:45:17,514::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:45:17,515::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:45:17,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:45:17,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:45:17,957::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:45:17,959::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 02:45:17,961::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 02:45:17,963::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 02:45:17,963::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 02:45:17,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:45:18,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:45:18,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:45:18,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00680437 s, 602 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:45:25,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:45:25,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109784 s, 373 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:45:28,416::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:45:28,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783712 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:45:35,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:45:35,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841565 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:45:38,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:45:38,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917809 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:45:45,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:45:45,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132826 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:45:48,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:45:48,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112343 s, 3.6 MB/s\n'; <rc> = 0
>Thread-14900::WARNING::2013-03-18 02:45:50,539::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:45:50,543::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:45:55,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:45:55,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153385 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:45:58,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:45:58,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077884 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:46:00,545::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:46:00,546::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:46:00,942::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:46:00,949::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:46:00,950::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:46:05,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:46:05,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178336 s, 230 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:46:08,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:46:08,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101897 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:46:15,576::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 02:46:15,578::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 02:46:15,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:46:15,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00126156 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:46:15,621::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00126156 s, 1.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 02:46:15,622::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 02:46:15,622::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 02:46:15,624::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 02:46:15,625::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:46:15,626::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:46:15,626::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:46:15,627::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:46:15,627::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:46:15,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:46:16,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:46:16,055::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:46:16,056::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 02:46:16,059::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 02:46:16,060::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 02:46:16,060::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 02:46:16,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:46:16,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:46:16,537::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:46:16,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00651116 s, 629 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:46:18,837::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:46:18,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135171 s, 303 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:46:26,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:46:26,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082671 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:46:28,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:46:28,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100291 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:46:36,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:46:36,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130609 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:46:38,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:46:39,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119859 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:46:46,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:46:46,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000725064 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:46:49,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:46:49,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834673 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:46:56,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:46:56,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117291 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:46:59,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:46:59,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119896 s, 3.4 MB/s\n'; <rc> = 0
>Thread-14901::WARNING::2013-03-18 02:47:00,976::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:47:00,979::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:47:06,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:47:06,766::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843945 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:47:09,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:47:09,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829722 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:47:10,984::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 02:47:10,985::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:47:10,986::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:47:11,445::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:47:11,451::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:47:11,453::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:47:16,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:47:16,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00308184 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:47:19,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:47:19,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113818 s, 360 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:47:26,814::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:47:26,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000979662 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:47:29,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:47:29,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150511 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:47:36,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:47:36,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136436 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:47:39,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:47:39,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180659 s, 227 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:47:46,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:47:46,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142626 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:47:49,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:47:49,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137845 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:47:56,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:47:56,979::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132022 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:47:59,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:47:59,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180614 s, 227 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:48:06,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:48:07,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775027 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:48:09,738::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:48:09,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760639 s, 5.4 MB/s\n'; <rc> = 0
>Thread-14902::WARNING::2013-03-18 02:48:11,481::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:48:11,486::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:48:17,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:48:17,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778415 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:48:19,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:48:19,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186126 s, 220 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:48:21,488::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:48:21,489::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:48:21,903::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:48:21,910::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:48:21,911::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:48:27,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:48:27,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00645331 s, 635 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:48:29,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:48:29,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013795 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:48:37,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:48:37,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013809 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:48:39,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:48:40,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00180948 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:48:47,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:48:47,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141308 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:48:50,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:48:50,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000750084 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:48:57,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:48:57,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000723936 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:49:00,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:49:00,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132452 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:49:07,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:49:07,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00505464 s, 810 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:49:10,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:49:10,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130835 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:49:17,332::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:49:17,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00278867 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:49:20,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:49:20,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122125 s, 3.4 MB/s\n'; <rc> = 0
>Thread-14903::WARNING::2013-03-18 02:49:21,935::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:49:21,939::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:49:27,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:49:27,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904377 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:49:30,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:49:30,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151631 s, 270 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:49:31,950::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:49:31,953::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:49:32,373::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:49:32,380::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:49:32,382::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:49:37,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:49:37,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868299 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:49:40,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:49:40,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00530408 s, 772 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:49:47,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:49:47,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142104 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:49:50,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:49:50,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149698 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:49:57,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:49:57,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158775 s, 258 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:50:00,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:50:00,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153318 s, 267 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:50:07,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:50:07,709::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107688 s, 380 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:50:10,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:50:10,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00262434 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:50:17,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:50:17,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000889527 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:50:20,837::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 02:50:20,838::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 02:50:20,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:50:20,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00127927 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:50:20,885::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00127927 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 02:50:20,886::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 02:50:20,886::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 02:50:20,888::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 02:50:20,889::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:50:20,889::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:50:20,890::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:50:20,890::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:50:20,890::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:50:20,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:50:21,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:50:21,335::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:50:21,336::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 02:50:21,338::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 02:50:21,340::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 02:50:21,340::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 02:50:21,341::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:50:21,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:50:21,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:50:21,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0050292 s, 814 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:50:27,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:50:27,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00329964 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:50:31,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:50:31,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102467 s, 4.0 MB/s\n'; <rc> = 0
>Thread-14904::WARNING::2013-03-18 02:50:32,422::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:50:32,426::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:50:37,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:50:37,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753942 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:50:42,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:50:42,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832243 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:50:42,428::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:50:42,430::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:50:42,839::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:50:42,846::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:50:42,847::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:50:47,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:50:47,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015026 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:50:52,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:50:52,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000695632 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:50:57,950::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:50:57,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838416 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:51:02,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:51:02,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825467 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:51:07,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:51:08,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.023973 s, 171 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:51:12,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:51:12,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125104 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:51:18,150::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 02:51:18,152::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 02:51:18,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:51:18,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00133942 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:51:18,203::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00133942 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 02:51:18,204::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 02:51:18,204::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 02:51:18,205::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 02:51:18,206::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:51:18,207::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:51:18,207::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:51:18,208::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:51:18,208::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:51:18,209::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:51:18,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:51:18,645::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:51:18,646::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 02:51:18,648::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 02:51:18,649::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 02:51:18,650::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 02:51:18,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:51:19,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:51:19,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:51:19,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011757 s, 348 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:51:22,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:51:22,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178326 s, 230 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:51:29,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:51:29,081::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737754 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:51:32,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:51:32,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0220153 s, 186 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:51:39,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:51:39,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901668 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:51:42,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:51:42,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00896944 s, 457 kB/s\n'; <rc> = 0
>Thread-14905::WARNING::2013-03-18 02:51:42,875::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:51:42,879::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:51:49,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:51:49,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877373 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:51:52,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:51:52,804::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100345 s, 408 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:51:52,884::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:51:52,885::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:51:53,412::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:51:53,419::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:51:53,420::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:51:59,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:51:59,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082731 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:52:02,863::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:52:02,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859969 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:52:09,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:52:09,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147558 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:52:12,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:52:12,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843348 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:52:19,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:52:19,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014251 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:52:22,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:52:23,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808132 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:52:29,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:52:29,331::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129041 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:52:33,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:52:33,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167721 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:52:39,341::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:52:39,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756012 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:52:43,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:52:43,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089411 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:52:49,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:52:49,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000726748 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:52:53,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:52:53,287::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000693863 s, 5.9 MB/s\n'; <rc> = 0
>Thread-14906::WARNING::2013-03-18 02:52:53,458::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:52:53,470::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 02:52:59,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:52:59,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136834 s, 3.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:53:03,473::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 02:53:03,474::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:53:03,475::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:53:03,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:53:03,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138794 s, 295 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:53:03,913::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:53:03,918::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:53:03,919::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:53:09,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:53:09,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859142 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:53:13,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:53:13,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822077 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:53:19,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:53:19,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000733152 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:53:23,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:53:23,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855012 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:53:29,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:53:29,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000703076 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:53:33,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:53:33,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0226156 s, 181 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:53:39,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:53:39,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00312933 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:53:43,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:53:43,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775684 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:53:49,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:53:49,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138184 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:53:53,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:53:53,973::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136739 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:53:59,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:53:59,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160683 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14907::WARNING::2013-03-18 02:54:03,946::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:54:03,951::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 02:54:04,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:54:04,054::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108254 s, 378 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:54:09,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:54:09,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00224621 s, 1.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:54:13,960::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:54:13,962::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:54:14,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:54:14,163::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00985264 s, 416 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:54:14,410::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:54:14,415::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:54:14,417::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:54:19,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:54:19,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00734125 s, 558 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:54:24,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:54:24,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824737 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:54:29,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:54:29,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150467 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:54:34,281::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:54:34,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874039 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:54:39,944::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:54:39,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000697267 s, 5.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:54:44,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:54:44,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150622 s, 272 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:54:49,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:54:50,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122523 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:54:54,445::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:54:54,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764188 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:55:00,019::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:55:00,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765698 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:55:04,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:55:04,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140123 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:55:10,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:55:10,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146878 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14908::WARNING::2013-03-18 02:55:14,444::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:55:14,448::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 02:55:14,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:55:14,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885322 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:55:20,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:55:20,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824582 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:55:24,451::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:55:24,452::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:55:24,666::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 02:55:24,669::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 02:55:24,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:55:24,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00977322 s, 210 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:55:24,730::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00977322 s, 210 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 02:55:24,730::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 02:55:24,730::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 02:55:24,732::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 02:55:24,734::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm reload operation' is holding the operation mutex, waiting...
>Thread-21::DEBUG::2013-03-18 02:55:24,924::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:55:24,930::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:55:24,930::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:55:24,960::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:55:24,964::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:55:24,974::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:55:24,975::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:55:24,975::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 02:55:24,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:55:25,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:55:25,450::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:55:25,451::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 02:55:25,454::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 02:55:25,455::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 02:55:25,455::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 02:55:25,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:55:25,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:55:25,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:55:25,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116367 s, 352 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:55:30,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:55:30,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154358 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:55:35,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:55:36,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150799 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:55:40,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:55:40,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00279423 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:55:46,053::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:55:46,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808481 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:55:50,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:55:50,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125464 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:55:56,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:55:56,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104647 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:56:00,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:56:00,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812533 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:56:06,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:56:06,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015278 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:56:10,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:56:10,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111756 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:56:16,270::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:56:16,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137071 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:56:20,391::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 02:56:20,395::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 02:56:20,396::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:56:20,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000790188 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:56:20,437::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000790188 s, 2.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 02:56:20,438::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 02:56:20,438::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 02:56:20,440::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 02:56:20,443::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:56:20,444::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:56:20,444::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:56:20,444::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:56:20,445::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 02:56:20,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:56:20,907::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:56:20,911::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 02:56:20,912::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 02:56:20,915::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 02:56:20,916::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 02:56:20,916::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 02:56:20,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:56:21,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:56:21,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:56:21,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125502 s, 326 kB/s\n'; <rc> = 0
>Thread-14909::WARNING::2013-03-18 02:56:24,971::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:56:24,978::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 02:56:26,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:56:26,401::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188436 s, 217 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:56:31,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:56:31,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848333 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:56:34,981::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:56:34,982::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:56:35,389::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:56:35,395::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:56:35,397::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:56:36,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:56:36,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135403 s, 303 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:56:41,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:56:41,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804869 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:56:46,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:56:46,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740191 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:56:51,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:56:51,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142501 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:56:56,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:56:56,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00543433 s, 754 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:57:01,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:57:01,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000670477 s, 6.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:57:06,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:57:06,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765101 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:57:11,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:57:11,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115161 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:57:16,740::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:57:16,770::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106048 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:57:21,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:57:21,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012631 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:57:26,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:57:26,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109362 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:57:31,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:57:31,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786858 s, 5.2 MB/s\n'; <rc> = 0
>Thread-14910::WARNING::2013-03-18 02:57:35,428::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:57:35,434::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 02:57:36,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:57:36,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120634 s, 340 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:57:41,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:57:41,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000910758 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:57:45,437::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:57:45,438::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:57:45,874::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:57:45,880::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:57:45,882::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:57:46,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:57:46,992::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00669087 s, 612 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:57:51,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:57:51,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765725 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:57:57,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:57:57,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801912 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:58:01,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:58:01,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920432 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:58:07,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:58:07,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838858 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:58:11,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:58:11,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892876 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:58:17,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:58:17,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808739 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:58:21,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:58:21,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0200486 s, 204 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:58:27,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:58:27,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000914966 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:58:31,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:58:31,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157178 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:58:37,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:58:37,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00908426 s, 451 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:58:41,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:58:41,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015158 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14911::WARNING::2013-03-18 02:58:45,919::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:58:45,924::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 02:58:47,429::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:58:47,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151274 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:58:51,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:58:51,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801752 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 02:58:55,925::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 02:58:55,926::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 02:58:55,927::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 02:58:56,408::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 02:58:56,414::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 02:58:56,416::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 02:58:57,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:58:57,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119511 s, 343 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:59:01,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:59:02,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113138 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:59:07,602::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:59:07,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792831 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:59:12,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:59:12,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862869 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:59:17,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:59:17,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089382 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:59:22,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:59:22,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141876 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:59:27,730::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:59:27,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000739501 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:59:32,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:59:32,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145931 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:59:37,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:59:37,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140172 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:59:42,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:59:42,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870114 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 02:59:47,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:59:47,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153688 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 02:59:52,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 02:59:52,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141563 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14912::WARNING::2013-03-18 02:59:56,440::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 02:59:56,444::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 02:59:57,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 02:59:58,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078755 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:00:02,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:00:02,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172617 s, 2.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:00:06,446::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:00:06,447::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:00:06,889::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:00:06,896::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:00:06,897::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:00:08,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:00:08,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016606 s, 247 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:00:12,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:00:12,395::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0286966 s, 143 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:00:18,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:00:18,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0385216 s, 106 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:00:22,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:00:22,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173498 s, 236 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:00:28,231::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 03:00:28,233::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 03:00:28,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:00:28,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000911563 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:00:28,265::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000911563 s, 2.2 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 03:00:28,265::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 03:00:28,266::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 03:00:28,268::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 03:00:28,269::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:00:28,270::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:00:28,271::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:00:28,271::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:00:28,272::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:00:28,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:00:28,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:00:28,695::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:00:28,696::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 03:00:28,698::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 03:00:28,699::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 03:00:28,700::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 03:00:28,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:00:29,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:00:29,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:00:29,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123544 s, 332 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:00:32,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:00:32,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00787416 s, 520 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:00:39,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:00:39,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000745428 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:00:42,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:00:42,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00756435 s, 541 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:00:49,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:00:49,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156147 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:00:52,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:00:52,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0030701 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:00:59,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:00:59,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139503 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:01:02,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:01:02,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120676 s, 3.4 MB/s\n'; <rc> = 0
>Thread-14913::WARNING::2013-03-18 03:01:06,928::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:01:06,932::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:01:09,439::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:01:09,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877625 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:01:12,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:01:12,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149191 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:01:16,935::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:01:16,936::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:01:17,343::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:01:17,349::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:01:17,350::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:01:19,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:01:19,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105392 s, 389 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:01:22,754::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 03:01:22,755::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 03:01:22,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:01:22,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000986854 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:01:22,783::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000986854 s, 2.1 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 03:01:22,784::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 03:01:22,784::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 03:01:22,785::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 03:01:22,787::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:01:22,787::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:01:22,788::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:01:22,788::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:01:22,788::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:01:22,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:01:23,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:01:23,197::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:01:23,198::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 03:01:23,200::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 03:01:23,202::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 03:01:23,202::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 03:01:23,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:01:23,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:01:23,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:01:23,622::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185253 s, 221 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:01:29,596::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:01:29,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136507 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:01:33,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:01:33,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171035 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:01:39,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:01:39,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000914844 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:01:43,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:01:43,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141033 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:01:49,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:01:49,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121563 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:01:53,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:01:53,739::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000671737 s, 6.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:01:59,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:01:59,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134429 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:02:03,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:02:03,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144091 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:02:09,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:02:09,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135705 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:02:13,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:02:13,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815715 s, 5.0 MB/s\n'; <rc> = 0
>Thread-14914::WARNING::2013-03-18 03:02:17,377::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:02:17,381::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:02:19,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:02:20,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904659 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:02:23,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:02:23,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860525 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:02:27,384::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:02:27,385::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:02:27,769::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:02:27,776::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:02:27,777::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:02:30,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:02:30,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00479684 s, 854 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:02:33,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:02:33,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000914527 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:02:40,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:02:40,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016171 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:02:43,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:02:43,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834632 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:02:50,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:02:50,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810205 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:02:53,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:02:53,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000724046 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:03:00,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:03:00,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145192 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:03:03,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:03:04,040::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106549 s, 384 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:03:10,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:03:10,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876049 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:03:14,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:03:14,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102189 s, 401 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:03:20,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:03:20,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846947 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:03:24,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:03:24,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013504 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14915::WARNING::2013-03-18 03:03:27,802::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:03:27,806::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:03:30,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:03:30,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798953 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:03:34,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:03:34,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837692 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:03:37,808::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:03:37,809::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:03:38,264::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:03:38,270::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:03:38,271::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:03:40,612::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:03:40,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130709 s, 313 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:03:44,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:03:44,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133386 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:03:50,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:03:50,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818385 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:03:54,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:03:54,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000705059 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:04:00,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:04:00,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863543 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:04:04,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:04:04,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000978781 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:04:10,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:04:10,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077835 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:04:14,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:04:14,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140896 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:04:20,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:04:20,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126361 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:04:24,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:04:24,404::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145957 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:04:30,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:04:31,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136179 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:04:34,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:04:34,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013373 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14916::WARNING::2013-03-18 03:04:38,299::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:04:38,303::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:04:41,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:04:41,081::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770951 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:04:44,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:04:44,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883819 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:04:48,305::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 03:04:48,306::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:04:48,306::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:04:48,741::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:04:48,748::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:04:48,750::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:04:51,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:04:51,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139458 s, 294 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:04:54,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:04:54,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143794 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:05:01,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:05:01,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844285 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:05:04,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:05:04,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112453 s, 364 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:05:11,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:05:11,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147386 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:05:14,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:05:14,636::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146169 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:05:21,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:05:21,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127908 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:05:24,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:05:24,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00286326 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:05:31,465::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 03:05:31,468::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 03:05:31,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:05:31,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00134307 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:05:31,515::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00134307 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 03:05:31,515::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 03:05:31,515::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 03:05:31,517::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 03:05:31,518::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:05:31,519::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:05:31,519::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:05:31,519::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:05:31,520::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:05:31,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:05:31,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:05:31,941::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:05:31,942::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 03:05:31,944::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 03:05:31,946::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 03:05:31,946::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 03:05:31,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:05:32,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:05:32,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:05:32,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01801 s, 227 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:05:34,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:05:34,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146048 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:05:42,416::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:05:42,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752274 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:05:44,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:05:44,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132372 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14917::WARNING::2013-03-18 03:05:48,780::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:05:48,786::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:05:52,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:05:52,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829616 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:05:54,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:05:54,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795635 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:05:58,794::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:05:58,795::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:05:59,251::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:05:59,257::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:05:59,258::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:06:02,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:06:02,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0072109 s, 568 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:06:04,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:06:04,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00573074 s, 715 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:06:12,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:06:12,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857027 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:06:14,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:06:14,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135743 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:06:22,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:06:22,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130354 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:06:24,969::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 03:06:24,970::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 03:06:24,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:06:24,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000990256 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:06:25,000::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000990256 s, 2.1 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 03:06:25,001::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 03:06:25,001::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 03:06:25,002::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 03:06:25,003::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:06:25,004::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:06:25,005::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:06:25,005::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:06:25,005::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:06:25,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:06:25,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:06:25,423::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:06:25,424::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 03:06:25,427::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 03:06:25,428::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 03:06:25,428::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 03:06:25,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:06:25,791::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:06:25,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:06:25,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017548 s, 233 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:06:32,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:06:32,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132195 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:06:35,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:06:35,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143883 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:06:42,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:06:42,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789352 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:06:45,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:06:45,950::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138613 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:06:52,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:06:53,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136156 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:06:55,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:06:55,992::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158251 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14918::WARNING::2013-03-18 03:06:59,288::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:06:59,296::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:07:03,070::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:07:03,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00221146 s, 1.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:07:05,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:07:06,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088381 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:07:09,300::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:07:09,301::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:07:09,784::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:07:09,790::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:07:09,792::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:07:13,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:07:13,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165695 s, 247 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:07:16,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:07:16,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835439 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:07:23,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:07:23,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102106 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:07:26,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:07:26,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790626 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:07:33,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:07:33,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142497 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:07:36,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:07:36,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122604 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:07:43,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:07:43,439::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129884 s, 315 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:07:46,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:07:46,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848533 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:07:53,482::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:07:53,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000929448 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:07:56,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:07:56,221::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130334 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:08:03,560::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:08:03,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154089 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:08:06,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:08:06,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796362 s, 5.1 MB/s\n'; <rc> = 0
>Thread-14919::WARNING::2013-03-18 03:08:09,826::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:08:09,833::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:08:13,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:08:13,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080675 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:08:16,271::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:08:16,297::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925989 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:08:19,836::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:08:19,837::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:08:20,285::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:08:20,291::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:08:20,293::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:08:23,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:08:23,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154121 s, 266 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:08:26,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:08:26,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000933778 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:08:33,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:08:33,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824479 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:08:36,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:08:36,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822385 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:08:43,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:08:43,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757187 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:08:46,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:08:46,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869982 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:08:53,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:08:53,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126809 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:08:56,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:08:56,448::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763252 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:09:04,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:09:04,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153739 s, 266 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:09:06,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:09:06,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161095 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:09:14,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:09:14,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851042 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:09:16,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:09:16,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00503469 s, 814 kB/s\n'; <rc> = 0
>Thread-14920::WARNING::2013-03-18 03:09:20,322::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:09:20,327::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:09:24,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:09:24,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00986889 s, 415 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:09:26,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:09:26,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815736 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:09:30,329::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:09:30,330::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:09:30,786::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:09:30,793::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:09:30,794::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:09:34,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:09:34,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128664 s, 318 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:09:36,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:09:36,617::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161017 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:09:44,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:09:44,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156416 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:09:46,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:09:46,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759042 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:09:54,455::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:09:54,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810522 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:09:56,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:09:56,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000695763 s, 5.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:10:04,524::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:10:04,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101128 s, 405 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:10:06,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:10:06,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.051889 s, 78.9 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:10:14,617::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:10:14,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826348 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:10:16,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:10:16,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102553 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:10:24,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:10:24,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145832 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:10:26,857::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:10:26,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00586608 s, 698 kB/s\n'; <rc> = 0
>Thread-14921::WARNING::2013-03-18 03:10:30,819::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:10:30,825::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:10:34,768::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 03:10:34,769::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 03:10:34,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:10:34,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00153103 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:10:34,814::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00153103 s, 1.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 03:10:34,815::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 03:10:34,815::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 03:10:34,816::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 03:10:34,817::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:10:34,818::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:10:34,818::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:10:34,819::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:10:34,819::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:10:34,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:10:35,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:10:35,252::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:10:35,253::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 03:10:35,255::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 03:10:35,256::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 03:10:35,257::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 03:10:35,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:10:35,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:10:35,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:10:35,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151806 s, 270 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:10:36,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:10:36,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112227 s, 3.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:10:40,833::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 03:10:40,834::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:10:40,835::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:10:41,234::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:10:41,240::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:10:41,242::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:10:45,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:10:45,756::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00900072 s, 455 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:10:46,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:10:47,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010102 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:10:55,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:10:55,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911588 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:10:57,021::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:10:57,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142989 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:11:05,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:11:05,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850653 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:11:07,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:11:07,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812174 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:11:15,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:11:15,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141064 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:11:17,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:11:17,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864263 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:11:26,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:11:26,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082513 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:11:27,161::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 03:11:27,165::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 03:11:27,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:11:27,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00154051 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:11:27,208::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00154051 s, 1.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 03:11:27,209::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 03:11:27,209::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 03:11:27,211::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 03:11:27,212::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:11:27,212::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:11:27,213::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:11:27,213::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:11:27,214::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:11:27,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:11:27,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:11:27,638::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:11:27,639::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 03:11:27,642::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 03:11:27,643::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 03:11:27,644::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 03:11:27,645::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:11:27,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:11:27,993::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:11:28,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155036 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:11:36,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:11:36,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866559 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:11:38,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:11:38,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146666 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14922::WARNING::2013-03-18 03:11:41,276::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:11:41,282::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:11:46,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:11:46,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807019 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:11:48,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:11:48,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131611 s, 3.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:11:51,285::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:11:51,286::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:11:51,723::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:11:51,729::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:11:51,731::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:11:56,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:11:56,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152504 s, 269 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:11:58,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:11:58,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120512 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:12:06,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:12:06,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150317 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:12:08,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:12:08,287::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000927094 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:12:16,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:12:16,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876577 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:12:18,292::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:12:18,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108437 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:12:26,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:12:26,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795191 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:12:28,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:12:28,404::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132704 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:12:36,560::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:12:36,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086462 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:12:38,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:12:38,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746487 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:12:46,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:12:46,671::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755559 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:12:48,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:12:48,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.002348 s, 1.7 MB/s\n'; <rc> = 0
>Thread-14923::WARNING::2013-03-18 03:12:51,766::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:12:51,774::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:12:56,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:12:56,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758922 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:12:58,528::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:12:58,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756455 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:13:01,834::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:13:01,835::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:13:02,274::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:13:02,281::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:13:02,282::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:13:06,792::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:13:06,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0220181 s, 186 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:13:08,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:13:08,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781182 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:13:16,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:13:16,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088142 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:13:18,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:13:18,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139629 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:13:26,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:13:26,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788895 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:13:28,648::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:13:28,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813962 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:13:37,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:13:37,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142621 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:13:38,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:13:38,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112866 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:13:47,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:13:47,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872431 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:13:48,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:13:48,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00294651 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:13:57,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:13:57,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136902 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:13:58,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:13:58,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140591 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14924::WARNING::2013-03-18 03:14:02,314::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:14:02,316::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:14:07,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:14:07,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881824 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:14:08,863::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:14:08,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861216 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:14:12,318::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:14:12,320::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:14:12,756::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:14:12,763::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:14:12,764::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:14:17,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:14:17,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00527201 s, 777 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:14:18,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:14:18,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809067 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:14:27,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:14:27,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923662 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:14:28,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:14:29,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103323 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:14:37,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:14:37,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150971 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:14:39,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:14:39,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103151 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:14:47,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:14:47,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823855 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:14:49,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:14:49,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138764 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:14:57,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:14:57,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137045 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:14:59,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:14:59,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000972602 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:15:07,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:15:07,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012816 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:15:09,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:15:09,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103925 s, 3.9 MB/s\n'; <rc> = 0
>Thread-14925::WARNING::2013-03-18 03:15:12,794::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:15:12,805::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:15:17,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:15:17,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0374001 s, 110 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:15:19,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:15:19,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104668 s, 3.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:15:22,810::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:15:22,812::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:15:23,286::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:15:23,292::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:15:23,294::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:15:27,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:15:27,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190759 s, 215 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:15:29,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:15:29,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768021 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:15:38,010::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 03:15:38,011::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 03:15:38,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:15:38,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00131219 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:15:38,057::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00131219 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 03:15:38,058::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 03:15:38,058::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 03:15:38,060::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 03:15:38,061::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:15:38,061::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:15:38,062::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:15:38,062::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:15:38,062::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:15:38,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:15:38,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:15:38,466::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:15:38,467::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 03:15:38,469::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 03:15:38,471::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 03:15:38,471::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 03:15:38,473::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:15:38,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:15:38,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:15:38,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00919274 s, 446 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:15:39,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:15:39,389::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000948651 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:15:48,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:15:48,989::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145527 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:15:49,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:15:49,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157445 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:15:59,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:15:59,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014379 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:15:59,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:15:59,478::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077633 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:16:09,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:16:09,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134333 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:16:09,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:16:09,548::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0051202 s, 800 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:16:19,221::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:16:19,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143577 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:16:19,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:16:19,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131065 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14926::WARNING::2013-03-18 03:16:23,322::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:16:23,324::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:16:29,305::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:16:29,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008017 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:16:29,622::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 03:16:29,624::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 03:16:29,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:16:29,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00147922 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:16:29,662::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00147922 s, 1.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 03:16:29,663::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 03:16:29,663::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 03:16:29,665::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 03:16:29,666::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:16:29,666::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:16:29,667::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:16:29,667::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:16:29,667::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:16:29,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:16:30,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:16:30,099::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:16:30,100::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 03:16:30,102::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 03:16:30,103::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 03:16:30,104::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 03:16:30,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:16:30,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:16:30,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:16:30,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136835 s, 299 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:16:33,326::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 03:16:33,327::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:16:33,328::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:16:33,749::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:16:33,755::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:16:33,757::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:16:39,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:16:39,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0342102 s, 120 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:16:40,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:16:40,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153446 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:16:49,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:16:49,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815983 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:16:50,560::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:16:50,590::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873032 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:16:59,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:16:59,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154267 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:17:00,596::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:17:00,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171493 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:17:09,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:17:09,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108679 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:17:10,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:17:10,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140591 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:17:19,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:17:19,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015056 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:17:20,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:17:20,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855755 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:17:29,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:17:29,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00189041 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:17:30,730::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:17:30,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130275 s, 3.1 MB/s\n'; <rc> = 0
>Thread-14927::WARNING::2013-03-18 03:17:33,781::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:17:33,784::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:17:39,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:17:39,928::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881094 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:17:40,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:17:40,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826725 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:17:43,787::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:17:43,788::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:17:44,193::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:17:44,199::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:17:44,200::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:17:49,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:17:50,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792908 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:17:50,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:17:50,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000989959 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:18:00,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:18:00,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145133 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:18:00,854::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:18:00,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087396 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:18:10,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:18:10,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817407 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:18:10,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:18:10,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000921806 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:18:20,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:18:20,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825622 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:18:20,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:18:20,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146029 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:18:30,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:18:30,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119939 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:18:30,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:18:31,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124084 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:18:40,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:18:40,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127233 s, 322 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:18:41,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:18:41,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138278 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14928::WARNING::2013-03-18 03:18:44,235::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:18:44,238::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:18:50,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:18:50,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112691 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:18:51,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:18:51,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859285 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:18:54,240::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:18:54,241::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:18:54,660::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:18:54,667::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:18:54,668::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:19:00,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:19:00,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814335 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:19:01,092::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:19:01,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862849 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:19:10,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:19:10,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139044 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:19:11,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:19:11,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151632 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:19:20,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:19:20,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126149 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:19:21,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:19:21,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137718 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:19:30,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:19:30,793::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147809 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:19:31,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:19:31,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747594 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:19:40,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:19:40,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140964 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:19:41,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:19:41,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142609 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:19:50,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:19:50,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837963 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:19:51,298::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:19:51,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136442 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14929::WARNING::2013-03-18 03:19:54,695::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:19:54,699::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:20:00,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:20:01,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834742 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:20:01,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:20:01,399::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0261583 s, 157 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:20:04,704::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:20:04,705::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:20:05,147::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:20:05,153::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:20:05,155::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:20:11,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:20:11,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186808 s, 219 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:20:11,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:20:11,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832461 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:20:21,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:20:21,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783451 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:20:21,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:20:21,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809796 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:20:31,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:20:31,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782336 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:20:31,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:20:31,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142843 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:20:41,329::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 03:20:41,330::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 03:20:41,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:20:41,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00117898 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:20:41,371::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00117898 s, 1.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 03:20:41,371::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 03:20:41,372::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 03:20:41,373::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 03:20:41,375::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:20:41,375::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:20:41,384::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:20:41,385::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:20:41,385::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:20:41,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:20:41,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:20:41,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138342 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:20:41,853::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:20:41,856::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:20:41,857::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 03:20:41,859::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 03:20:41,860::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 03:20:41,860::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 03:20:41,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:20:42,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:20:42,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:20:42,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0081769 s, 501 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:20:51,596::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:20:51,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793702 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:20:52,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:20:52,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135328 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:21:01,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:21:01,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148342 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:21:02,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:21:02,456::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150197 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14930::WARNING::2013-03-18 03:21:05,178::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:21:05,182::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 03:21:11,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:21:11,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826798 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:21:12,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:21:12,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112931 s, 3.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:21:15,184::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:21:15,185::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:21:15,622::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:21:15,630::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:21:15,632::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:21:21,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:21:21,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000958921 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:21:22,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:21:22,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147125 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:21:31,753::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 03:21:31,755::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 03:21:31,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:21:31,780::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000751347 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:21:31,787::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000751347 s, 2.7 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 03:21:31,788::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 03:21:31,788::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 03:21:31,790::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 03:21:31,791::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:21:31,791::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:21:31,792::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:21:31,792::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:21:31,792::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:21:31,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:21:32,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:21:32,229::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:21:32,230::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 03:21:32,232::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 03:21:32,233::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 03:21:32,234::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 03:21:32,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:21:32,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:21:32,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:21:32,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:21:32,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149776 s, 273 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:21:32,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158613 s, 258 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:21:42,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:21:42,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140656 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:21:42,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:21:42,805::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151346 s, 271 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:21:52,741::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:21:52,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078864 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:21:52,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:21:52,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776644 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:22:02,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:22:02,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0267191 s, 153 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:22:02,910::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:22:02,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074552 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:22:12,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:22:12,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019178 s, 214 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:22:12,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:22:13,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177076 s, 231 kB/s\n'; <rc> = 0
>Thread-14931::WARNING::2013-03-18 03:22:15,658::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:22:15,664::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 03:22:22,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:22:22,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859554 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:22:23,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:22:23,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886379 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:22:25,667::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 03:22:25,668::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:22:25,669::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:22:26,125::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:22:26,132::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:22:26,133::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:22:32,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:22:32,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873754 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:22:33,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:22:33,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123126 s, 333 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:22:42,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:22:43,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855197 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:22:43,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:22:43,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000977087 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:22:53,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:22:53,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133336 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:22:53,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:22:53,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143144 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:23:03,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:23:03,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147468 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:23:03,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:23:03,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140678 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:23:13,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:23:13,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886291 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:23:13,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:23:13,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102879 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:23:23,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:23:23,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818327 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:23:23,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:23:23,593::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161443 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14932::WARNING::2013-03-18 03:23:26,171::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:23:26,175::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 03:23:33,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:23:33,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805387 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:23:33,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:23:33,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917502 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:23:36,178::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:23:36,179::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:23:36,625::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:23:36,634::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:23:36,635::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:23:43,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:23:43,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00192947 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:23:43,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:23:43,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862903 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:23:53,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:23:53,297::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846288 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:23:53,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:23:53,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0214984 s, 191 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:24:03,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:24:03,331::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091406 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:24:03,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:24:03,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829356 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:24:13,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:24:13,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846073 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:24:13,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:24:13,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147529 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:24:23,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:24:23,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158144 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:24:24,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:24:24,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111954 s, 366 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:24:33,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:24:33,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148764 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:24:34,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:24:34,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115238 s, 3.6 MB/s\n'; <rc> = 0
>Thread-14933::WARNING::2013-03-18 03:24:36,662::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:24:36,668::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 03:24:43,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:24:43,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000944476 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:24:44,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:24:44,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100725 s, 4.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:24:46,671::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:24:46,672::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:24:47,145::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:24:47,152::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:24:47,153::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:24:53,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:24:53,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150989 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:24:54,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:24:54,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849801 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:25:03,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:25:03,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130471 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:25:04,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:25:04,358::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.012146 s, 337 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:25:13,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:25:13,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907901 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:25:14,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:25:14,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0411142 s, 99.6 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:25:23,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:25:23,649::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816407 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:25:24,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:25:24,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000745911 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:25:33,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:25:33,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0210091 s, 195 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:25:34,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:25:34,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159024 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:25:43,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:25:43,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00189323 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:25:44,692::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 03:25:44,697::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 03:25:44,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:25:44,733::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00130391 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:25:44,739::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00130391 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 03:25:44,739::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 03:25:44,739::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 03:25:44,746::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 03:25:44,747::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:25:44,748::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:25:44,748::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:25:44,749::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:25:44,749::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:25:44,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:25:45,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:25:45,202::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:25:45,211::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 03:25:45,238::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 03:25:45,241::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 03:25:45,242::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 03:25:45,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:25:45,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:25:45,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:25:45,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00849379 s, 482 kB/s\n'; <rc> = 0
>Thread-14934::WARNING::2013-03-18 03:25:47,182::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:25:47,188::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 03:25:53,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:25:53,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175416 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:25:55,863::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:25:55,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0216854 s, 189 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:25:57,191::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:25:57,192::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:25:57,612::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:25:57,619::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:25:57,620::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:26:03,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:26:03,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146162 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:26:05,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:26:05,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092389 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:26:13,863::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:26:13,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082774 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:26:16,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:26:16,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766515 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:26:23,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:26:23,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811961 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:26:26,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:26:26,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143938 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:26:33,956::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 03:26:33,957::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 03:26:33,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:26:34,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00141673 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:26:34,008::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00141673 s, 1.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 03:26:34,008::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 03:26:34,008::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 03:26:34,010::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 03:26:34,011::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:26:34,012::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:26:34,012::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:26:34,012::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:26:34,013::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:26:34,014::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:26:34,413::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:26:34,417::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:26:34,418::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 03:26:34,420::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 03:26:34,421::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 03:26:34,422::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 03:26:34,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:26:34,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:26:34,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:26:34,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181899 s, 225 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:26:36,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:26:36,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189918 s, 216 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:26:44,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:26:44,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00313647 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:26:46,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:26:46,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125533 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:26:54,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:26:54,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857225 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:26:56,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:26:56,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138776 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14935::WARNING::2013-03-18 03:26:57,646::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:26:57,650::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 03:27:04,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:27:04,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000908928 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:27:06,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:27:06,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884165 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:27:07,652::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:27:07,653::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:27:08,130::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:27:08,137::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:27:08,138::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:27:14,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:27:15,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770494 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:27:16,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:27:16,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894926 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:27:25,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:27:25,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146748 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:27:26,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:27:26,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827009 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:27:35,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:27:35,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142791 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:27:36,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:27:37,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159109 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:27:45,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:27:45,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142526 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:27:47,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:27:47,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135333 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:27:55,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:27:55,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146293 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:27:57,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:27:57,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146275 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:28:05,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:28:05,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118362 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:28:07,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:28:07,402::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0305557 s, 134 kB/s\n'; <rc> = 0
>Thread-14936::WARNING::2013-03-18 03:28:08,179::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:28:08,186::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 03:28:15,254::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:28:15,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765407 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:28:17,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:28:17,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105422 s, 389 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:28:18,195::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 03:28:18,197::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:28:18,199::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:28:18,594::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:28:18,600::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:28:18,602::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:28:25,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:28:25,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140904 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:28:27,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:28:27,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843421 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:28:35,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:28:35,365::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139588 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:28:37,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:28:37,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000921167 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:28:45,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:28:45,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126392 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:28:47,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:28:47,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131666 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:28:55,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:28:55,448::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136948 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:28:58,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:28:58,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00215101 s, 1.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:29:05,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:29:05,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012894 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:29:08,102::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:29:08,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001373 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:29:15,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:29:15,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103011 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:29:18,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:29:18,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109647 s, 3.7 MB/s\n'; <rc> = 0
>Thread-14937::WARNING::2013-03-18 03:29:18,633::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:29:18,637::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 03:29:25,556::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:29:25,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104378 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:29:28,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:29:28,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109007 s, 376 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:29:28,639::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:29:28,640::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:29:29,073::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:29:29,079::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:29:29,080::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:29:35,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:29:35,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158433 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:29:38,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:29:38,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091111 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:29:45,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:29:45,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156733 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:29:48,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:29:48,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000971885 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:29:55,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:29:55,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00438144 s, 935 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:29:58,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:29:58,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751938 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:30:05,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:30:05,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132546 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:30:08,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:30:08,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00951 s, 431 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:30:15,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:30:15,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137226 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:30:18,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:30:18,912::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00876009 s, 468 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:30:25,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:30:25,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104784 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:30:28,952::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:30:28,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774979 s, 5.3 MB/s\n'; <rc> = 0
>Thread-14938::WARNING::2013-03-18 03:30:29,121::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:30:29,136::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 03:30:35,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:30:35,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000926478 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:30:39,144::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:30:39,145::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:30:39,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:30:39,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124336 s, 329 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:30:39,583::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:30:39,588::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:30:39,590::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:30:45,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:30:45,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172813 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:30:49,290::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 03:30:49,291::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 03:30:49,292::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:30:49,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0167054 s, 123 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:30:49,337::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0167054 s, 123 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 03:30:49,338::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 03:30:49,338::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 03:30:49,340::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 03:30:49,342::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:30:49,342::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:30:49,343::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:30:49,344::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:30:49,344::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:30:49,345::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:30:49,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:30:49,771::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:30:49,772::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 03:30:49,775::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 03:30:49,776::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 03:30:49,777::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 03:30:49,778::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:30:50,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:30:50,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:30:50,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186508 s, 220 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:30:55,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:30:55,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145173 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:31:00,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:31:00,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749278 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:31:05,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:31:06,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819164 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:31:10,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:31:10,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111074 s, 369 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:31:16,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:31:16,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825243 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:31:20,405::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:31:20,450::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138826 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:31:26,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:31:26,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136862 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:31:30,492::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:31:30,549::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117489 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:31:36,093::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 03:31:36,095::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 03:31:36,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:31:36,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000721133 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:31:36,129::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000721133 s, 2.8 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 03:31:36,132::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 03:31:36,133::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 03:31:36,135::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 03:31:36,136::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:31:36,138::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:31:36,139::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:31:36,139::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:31:36,139::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:31:36,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:31:36,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:31:36,584::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:31:36,586::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 03:31:36,588::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 03:31:36,590::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 03:31:36,590::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 03:31:36,592::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:31:36,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:31:36,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:31:37,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112242 s, 365 kB/s\n'; <rc> = 0
>Thread-14939::WARNING::2013-03-18 03:31:39,619::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:31:39,625::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:31:40,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:31:40,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121431 s, 337 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:31:47,014::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:31:47,060::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017842 s, 2.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:31:49,628::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:31:49,629::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:31:50,056::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:31:50,063::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:31:50,064::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:31:50,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:31:50,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109644 s, 374 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:31:57,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:31:57,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166063 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:32:00,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:32:00,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837243 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:32:07,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:32:07,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913868 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:32:10,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:32:10,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148299 s, 276 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:32:17,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:32:17,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810187 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:32:20,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:32:20,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900907 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:32:27,194::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:32:27,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846022 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:32:30,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:32:31,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133955 s, 306 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:32:37,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:32:37,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832978 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:32:41,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:32:41,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765739 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:32:47,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:32:47,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793531 s, 5.2 MB/s\n'; <rc> = 0
>Thread-14940::WARNING::2013-03-18 03:32:50,094::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:32:50,103::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:32:51,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:32:51,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139229 s, 294 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:32:57,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:32:57,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00435267 s, 941 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:33:00,107::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:33:00,108::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:33:00,526::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:33:00,533::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:33:00,534::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:33:01,230::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:33:01,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0200057 s, 205 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:33:07,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:33:07,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00209854 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:33:11,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:33:11,358::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136071 s, 301 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:33:17,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:33:17,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808852 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:33:21,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:33:21,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787613 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:33:27,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:33:27,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147801 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:33:31,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:33:31,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017126 s, 239 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:33:37,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:33:37,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130509 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:33:41,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:33:41,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077934 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:33:47,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:33:47,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139577 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:33:51,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:33:51,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015284 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:33:57,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:33:57,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000956083 s, 4.3 MB/s\n'; <rc> = 0
>Thread-14941::WARNING::2013-03-18 03:34:00,563::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:34:00,576::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:34:01,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:34:01,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763247 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:34:07,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:34:07,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917205 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:34:10,579::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 03:34:10,580::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:34:10,581::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:34:11,016::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:34:11,022::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:34:11,024::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:34:11,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:34:11,866::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135958 s, 301 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:34:17,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:34:17,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001215 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:34:21,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:34:21,935::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821635 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:34:27,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:34:27,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827916 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:34:31,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:34:32,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00529592 s, 773 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:34:37,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:34:37,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139942 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:34:42,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:34:42,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751744 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:34:47,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:34:47,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0070642 s, 580 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:34:52,128::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:34:52,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152741 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:34:57,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:34:57,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00539197 s, 760 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:35:02,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:35:02,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850968 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:35:07,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:35:07,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141118 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14942::WARNING::2013-03-18 03:35:11,052::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:35:11,057::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:35:12,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:35:12,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100956 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:35:17,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:35:17,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810756 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:35:21,059::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:35:21,060::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:35:21,506::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:35:21,512::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:35:21,514::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:35:22,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:35:22,404::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139693 s, 293 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:35:27,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:35:28,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168794 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:35:32,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:35:32,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0289337 s, 142 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:35:38,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:35:38,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164242 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:35:42,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:35:42,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149471 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:35:48,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:35:48,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00348177 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:35:52,639::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 03:35:52,640::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 03:35:52,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:35:52,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00117647 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:35:52,684::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00117647 s, 1.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 03:35:52,685::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 03:35:52,685::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 03:35:52,687::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 03:35:52,688::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:35:52,688::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:35:52,689::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:35:52,689::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:35:52,690::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:35:52,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:35:53,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:35:53,157::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:35:53,158::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 03:35:53,161::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 03:35:53,162::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 03:35:53,162::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 03:35:53,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:35:53,521::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:35:53,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:35:53,562::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00635342 s, 645 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:35:58,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:35:58,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171314 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:36:03,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:36:03,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128292 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:36:08,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:36:08,238::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199374 s, 205 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:36:13,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:36:13,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136892 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:36:18,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:36:18,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157641 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14943::WARNING::2013-03-18 03:36:21,548::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:36:21,552::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:36:23,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:36:23,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0400389 s, 102 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:36:28,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:36:28,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787518 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:36:31,562::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:36:31,563::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:36:31,995::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:36:32,002::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:36:32,003::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:36:33,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:36:33,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00741597 s, 552 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:36:38,338::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 03:36:38,340::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 03:36:38,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:36:38,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00114797 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:36:38,387::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00114797 s, 1.8 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 03:36:38,387::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 03:36:38,388::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 03:36:38,389::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 03:36:38,390::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:36:38,391::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:36:38,391::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:36:38,392::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:36:38,392::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:36:38,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:36:38,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:36:38,813::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:36:38,814::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 03:36:38,816::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 03:36:38,817::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 03:36:38,818::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 03:36:38,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:36:39,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:36:39,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:36:39,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150907 s, 271 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:36:43,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:36:44,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0511448 s, 80.1 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:36:49,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:36:49,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137085 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:36:54,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:36:54,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854939 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:36:59,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:36:59,333::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142058 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:37:04,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:37:04,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0305736 s, 134 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:37:09,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:37:09,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848261 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:37:14,242::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:37:14,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847911 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:37:19,373::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:37:19,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000956616 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:37:24,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:37:24,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00873366 s, 469 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:37:29,416::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:37:29,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082635 s, 5.0 MB/s\n'; <rc> = 0
>Thread-14944::WARNING::2013-03-18 03:37:32,034::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:37:32,038::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:37:34,395::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:37:34,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824804 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:37:39,465::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:37:39,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150352 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:37:42,040::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:37:42,041::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:37:42,517::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:37:42,523::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:37:42,524::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:37:44,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:37:44,517::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149528 s, 274 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:37:49,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:37:49,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014618 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:37:54,556::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:37:54,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811686 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:37:59,560::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:37:59,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000723963 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:38:04,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:38:04,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0223816 s, 183 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:38:09,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:38:09,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157815 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:38:14,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:38:14,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758965 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:38:19,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:38:19,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100943 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:38:24,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:38:24,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00188446 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:38:29,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:38:29,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850523 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:38:34,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:38:34,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017958 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:38:39,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:38:39,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920444 s, 4.5 MB/s\n'; <rc> = 0
>Thread-14945::WARNING::2013-03-18 03:38:42,549::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:38:42,553::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:38:44,955::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:38:44,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00388852 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:38:49,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:38:49,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158939 s, 2.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:38:52,555::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:38:52,557::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:38:52,963::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:38:52,969::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:38:52,971::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:38:55,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:38:55,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126352 s, 324 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:38:59,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:38:59,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793594 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:39:05,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:39:05,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882612 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:39:09,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:39:09,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150275 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:39:15,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:39:15,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000702768 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:39:19,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:39:19,958::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000737957 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:39:25,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:39:25,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136735 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:39:29,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:39:30,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127628 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:39:35,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:39:35,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818134 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:39:40,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:39:40,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788263 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:39:45,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:39:45,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00251711 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:39:50,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:39:50,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116596 s, 3.5 MB/s\n'; <rc> = 0
>Thread-14946::WARNING::2013-03-18 03:39:52,997::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:39:53,003::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:39:55,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:39:55,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146839 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:40:00,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:40:00,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008638 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:40:03,006::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 03:40:03,007::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:40:03,008::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:40:03,438::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:40:03,445::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:40:03,447::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:40:05,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:40:05,598::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00695777 s, 589 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:40:10,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:40:10,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150602 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:40:15,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:40:15,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774698 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:40:20,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:40:20,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899925 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:40:25,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:40:25,765::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154138 s, 266 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:40:30,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:40:30,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000717608 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:40:35,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:40:35,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136304 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:40:40,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:40:40,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0054862 s, 747 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:40:45,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:40:45,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139032 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:40:50,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:40:50,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.020237 s, 202 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:40:55,956::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 03:40:55,960::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 03:40:55,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:40:55,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00128745 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:40:56,001::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00128745 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 03:40:56,001::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 03:40:56,002::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 03:40:56,003::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 03:40:56,004::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:40:56,005::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:40:56,005::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:40:56,006::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:40:56,006::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:40:56,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:40:56,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:40:56,441::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:40:56,442::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 03:40:56,444::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 03:40:56,445::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 03:40:56,446::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 03:40:56,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:40:56,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:40:56,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:40:56,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00832026 s, 492 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:41:00,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:41:00,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000957692 s, 4.3 MB/s\n'; <rc> = 0
>Thread-14947::WARNING::2013-03-18 03:41:03,471::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:41:03,475::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:41:06,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:41:06,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817858 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:41:10,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:41:10,494::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0088498 s, 463 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:41:13,477::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:41:13,478::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:41:13,924::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:41:13,931::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:41:13,932::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:41:17,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:41:17,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142427 s, 288 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:41:20,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:41:20,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082024 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:41:27,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:41:27,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142849 s, 287 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:41:30,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:41:30,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000716961 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:41:37,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:41:37,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831329 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:41:40,583::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 03:41:40,585::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 03:41:40,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:41:40,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000671806 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:41:40,625::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000671806 s, 3.0 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 03:41:40,625::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 03:41:40,625::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 03:41:40,627::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 03:41:40,628::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:41:40,628::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:41:40,629::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:41:40,629::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:41:40,630::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:41:40,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:41:41,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:41:41,063::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:41:41,064::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 03:41:41,066::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 03:41:41,067::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 03:41:41,068::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 03:41:41,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:41:41,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:41:41,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:41:41,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138508 s, 296 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:41:47,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:41:47,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145931 s, 281 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:41:51,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:41:51,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135695 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:41:57,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:41:57,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144309 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:42:01,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:42:01,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014667 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:42:07,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:42:07,483::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805893 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:42:11,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:42:11,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897333 s, 4.6 MB/s\n'; <rc> = 0
>Thread-14948::WARNING::2013-03-18 03:42:13,963::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:42:13,970::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:42:17,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:42:17,562::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788553 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:42:21,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:42:21,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773233 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:42:23,973::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:42:23,974::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:42:24,432::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:42:24,440::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:42:24,442::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:42:27,605::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:42:27,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149612 s, 274 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:42:31,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:42:31,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754565 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:42:37,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:42:37,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821709 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:42:41,701::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:42:41,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819756 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:42:47,769::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:42:47,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847929 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:42:51,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:42:51,765::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901028 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:42:57,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:42:57,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137052 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:43:01,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:43:01,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152203 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:43:07,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:43:07,970::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100016 s, 410 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:43:11,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:43:11,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886151 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:43:18,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:43:18,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00459003 s, 892 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:43:21,854::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:43:21,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014947 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14949::WARNING::2013-03-18 03:43:24,466::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:43:24,474::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:43:28,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:43:28,130::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0077596 s, 528 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:43:31,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:43:31,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809272 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:43:34,477::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:43:34,478::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:43:34,892::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:43:34,900::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:43:34,901::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:43:38,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:43:38,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0221055 s, 185 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:43:41,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:43:41,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886286 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:43:48,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:43:48,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127074 s, 322 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:43:51,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:43:52,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833622 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:43:58,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:43:58,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800936 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:44:02,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:44:02,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132278 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:44:08,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:44:08,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00855327 s, 479 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:44:12,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:44:12,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151066 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:44:18,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:44:18,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000725045 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:44:22,108::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:44:22,140::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00302902 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:44:28,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:44:28,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757749 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:44:32,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:44:32,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162684 s, 2.5 MB/s\n'; <rc> = 0
>Thread-14950::WARNING::2013-03-18 03:44:34,931::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:44:34,936::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:44:38,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:44:38,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000724528 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:44:42,196::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:44:42,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836552 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:44:44,938::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:44:44,939::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:44:45,437::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:44:45,444::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:44:45,445::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:44:48,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:44:48,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180703 s, 227 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:44:52,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:44:52,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000948481 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:44:58,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:44:58,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000721195 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:45:02,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:45:02,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117253 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:45:08,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:45:08,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00791421 s, 518 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:45:12,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:45:12,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800714 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:45:18,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:45:18,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754565 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:45:22,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:45:22,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897157 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:45:29,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:45:29,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159513 s, 257 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:45:32,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:45:32,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00210375 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:45:39,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:45:39,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844978 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:45:42,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:45:42,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825703 s, 5.0 MB/s\n'; <rc> = 0
>Thread-14951::WARNING::2013-03-18 03:45:45,473::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:45:45,477::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:45:49,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:45:49,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747137 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:45:52,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:45:52,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113164 s, 3.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:45:55,479::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 03:45:55,480::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:45:55,481::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:45:55,900::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:45:55,907::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:45:55,908::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:45:59,274::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 03:45:59,280::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 03:45:59,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:45:59,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0108191 s, 189 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:45:59,324::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0108191 s, 189 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 03:45:59,324::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 03:45:59,324::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 03:45:59,326::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 03:45:59,327::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:45:59,328::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:45:59,328::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:45:59,328::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:45:59,329::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:45:59,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:45:59,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:45:59,754::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:45:59,755::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 03:45:59,757::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 03:45:59,759::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 03:45:59,759::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 03:45:59,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:46:00,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:46:00,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:46:00,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176443 s, 232 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:46:02,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:46:02,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000989628 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:46:10,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:46:10,287::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188104 s, 218 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:46:12,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:46:12,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145446 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:46:20,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:46:20,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166847 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:46:22,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:46:22,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109943 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:46:30,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:46:30,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00097589 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:46:32,648::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:46:32,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178889 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:46:40,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:46:40,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872214 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:46:42,695::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 03:46:42,696::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 03:46:42,697::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:46:42,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00115914 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:46:42,777::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00115914 s, 1.8 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 03:46:42,777::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 03:46:42,778::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 03:46:42,780::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 03:46:42,781::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:46:42,781::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:46:42,782::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:46:42,782::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:46:42,783::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:46:42,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:46:43,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:46:43,260::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:46:43,260::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 03:46:43,263::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 03:46:43,264::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 03:46:43,265::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 03:46:43,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:46:43,647::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:46:43,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:46:43,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00667844 s, 613 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:46:50,544::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:46:50,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0073428 s, 558 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:46:53,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:46:53,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149095 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14952::WARNING::2013-03-18 03:46:55,934::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:46:55,938::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:47:00,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:47:00,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831104 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:47:03,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:47:03,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875061 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:47:05,940::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:47:05,941::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:47:06,363::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:47:06,370::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:47:06,371::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:47:10,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:47:10,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00432122 s, 948 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:47:13,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:47:13,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133017 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:47:20,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:47:20,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788847 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:47:23,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:47:23,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128462 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:47:30,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:47:30,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875847 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:47:33,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:47:33,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146462 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:47:40,938::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:47:40,979::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134796 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:47:43,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:47:43,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131909 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:47:51,021::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:47:51,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773716 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:47:53,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:47:53,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768612 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:48:01,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:48:01,130::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013063 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:48:04,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:48:04,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000903457 s, 4.5 MB/s\n'; <rc> = 0
>Thread-14953::WARNING::2013-03-18 03:48:06,397::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:48:06,401::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:48:11,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:48:11,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000953478 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:48:14,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:48:14,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146497 s, 2.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:48:16,403::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:48:16,404::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:48:16,826::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:48:16,833::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:48:16,834::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:48:21,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:48:21,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142933 s, 287 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:48:24,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:48:24,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107507 s, 381 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:48:31,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:48:31,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763539 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:48:34,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:48:34,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000960599 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:48:41,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:48:41,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00232351 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:48:44,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:48:44,222::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00342697 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:48:51,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:48:51,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143441 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:48:54,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:48:54,260::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790315 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:49:01,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:49:01,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159027 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:49:04,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:49:04,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741965 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:49:11,614::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:49:11,657::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141731 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:49:14,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:49:14,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00342578 s, 1.2 MB/s\n'; <rc> = 0
>Thread-14954::WARNING::2013-03-18 03:49:16,861::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:49:16,867::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:49:21,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:49:21,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785749 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:49:24,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:49:24,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865416 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:49:26,871::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:49:26,872::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:49:27,319::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:49:27,325::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:49:27,327::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:49:31,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:49:31,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158907 s, 258 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:49:34,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:49:34,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00571563 s, 717 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:49:41,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:49:41,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794933 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:49:44,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:49:44,456::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738973 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:49:51,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:49:51,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168249 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:49:54,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:49:54,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839263 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:50:02,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:50:02,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00237068 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:50:04,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:50:04,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728875 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:50:12,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:50:12,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836113 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:50:14,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:50:14,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00842105 s, 486 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:50:22,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:50:22,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137764 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:50:24,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:50:24,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126142 s, 3.2 MB/s\n'; <rc> = 0
>Thread-14955::WARNING::2013-03-18 03:50:27,355::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:50:27,363::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:50:32,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:50:32,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776269 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:50:34,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:50:34,765::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781396 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:50:37,365::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:50:37,366::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:50:37,796::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:50:37,802::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:50:37,804::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:50:42,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:50:42,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192612 s, 213 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:50:44,782::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:50:44,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153726 s, 266 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:50:52,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:50:52,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803994 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:50:54,863::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:50:54,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795348 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:51:02,486::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 03:51:02,487::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 03:51:02,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:51:02,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000794988 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:51:02,515::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000794988 s, 2.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 03:51:02,516::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 03:51:02,516::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 03:51:02,518::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 03:51:02,519::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:51:02,519::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:51:02,520::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:51:02,520::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:51:02,521::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:51:02,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:51:02,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:51:02,967::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:51:02,968::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 03:51:02,970::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 03:51:02,971::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 03:51:02,972::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 03:51:02,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:51:03,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:51:03,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:51:03,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00410209 s, 999 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:51:04,933::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:51:05,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00951611 s, 430 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:51:13,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:51:13,483::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146027 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:51:15,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:51:15,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124791 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:51:23,523::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:51:23,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00833758 s, 491 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:51:25,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:51:25,175::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00781948 s, 524 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:51:33,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:51:33,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144042 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:51:35,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:51:35,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00300809 s, 1.4 MB/s\n'; <rc> = 0
>Thread-14956::WARNING::2013-03-18 03:51:37,831::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:51:37,835::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:51:43,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:51:43,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00627016 s, 653 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:51:45,274::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 03:51:45,275::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 03:51:45,277::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:51:45,318::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000888859 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:51:45,321::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000888859 s, 2.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 03:51:45,322::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 03:51:45,322::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 03:51:45,331::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 03:51:45,332::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:51:45,333::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:51:45,334::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:51:45,334::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:51:45,335::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:51:45,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:51:45,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:51:45,832::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:51:45,833::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 03:51:45,835::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 03:51:45,844::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 03:51:45,844::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 03:51:45,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:51:46,299::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:51:46,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:51:46,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163149 s, 251 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:51:47,842::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 03:51:47,843::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:51:47,844::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:51:48,272::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:51:48,279::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:51:48,280::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:51:53,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:51:53,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00737747 s, 555 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:51:56,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:51:56,392::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159167 s, 257 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:52:03,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:52:03,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800849 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:52:06,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:52:06,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161077 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:52:13,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:52:13,944::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867346 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:52:16,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:52:16,483::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899422 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:52:23,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:52:24,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00973992 s, 421 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:52:26,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:52:26,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0051793 s, 791 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:52:34,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:52:34,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133903 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:52:36,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:52:36,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.02586 s, 158 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:52:44,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:52:44,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137453 s, 298 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:52:46,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:52:46,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793478 s, 5.2 MB/s\n'; <rc> = 0
>Thread-14957::WARNING::2013-03-18 03:52:48,318::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:52:48,324::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:52:54,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:52:54,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783728 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:52:56,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:52:56,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085762 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:52:58,327::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:52:58,328::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:52:58,752::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:52:58,758::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:52:58,759::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:53:04,312::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:53:04,358::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152401 s, 269 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:53:06,778::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:53:06,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814073 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:53:14,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:53:14,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864712 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:53:16,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:53:16,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758414 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:53:24,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:53:24,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0223242 s, 183 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:53:26,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:53:26,979::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00842558 s, 486 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:53:34,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:53:34,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000734307 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:53:36,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:53:37,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141161 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:53:44,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:53:44,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783963 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:53:47,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:53:47,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149198 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:53:54,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:53:54,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158343 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:53:57,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:53:57,120::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000968351 s, 4.2 MB/s\n'; <rc> = 0
>Thread-14958::WARNING::2013-03-18 03:53:58,794::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:53:58,798::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:54:04,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:54:04,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836187 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:54:07,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:54:07,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0370504 s, 111 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:54:08,800::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:54:08,802::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:54:09,269::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:54:09,275::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:54:09,276::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:54:14,861::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:54:14,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00818513 s, 500 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:54:17,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:54:17,260::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014088 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:54:24,941::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:54:24,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803902 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:54:27,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:54:27,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149776 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:54:35,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:54:35,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894245 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:54:37,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:54:37,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789113 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:54:45,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:54:45,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775781 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:54:47,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:54:47,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882597 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:54:55,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:54:55,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124087 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:54:57,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:54:57,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868663 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:55:05,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:55:05,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128401 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:55:07,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:55:07,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116525 s, 3.5 MB/s\n'; <rc> = 0
>Thread-14959::WARNING::2013-03-18 03:55:09,316::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:55:09,322::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:55:15,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:55:15,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109464 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:55:17,471::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:55:17,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801983 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:55:19,331::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:55:19,332::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:55:19,766::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:55:19,772::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:55:19,773::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:55:25,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:55:25,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016174 s, 253 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:55:27,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:55:27,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858847 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:55:35,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:55:35,513::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821197 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:55:37,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:55:37,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875826 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:55:45,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:55:45,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168443 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:55:47,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:55:47,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138111 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:55:55,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:55:55,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128353 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:55:57,615::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:55:57,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846328 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:56:05,742::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 03:56:05,744::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 03:56:05,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:56:05,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000754051 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:56:05,773::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000754051 s, 2.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 03:56:05,774::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 03:56:05,774::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 03:56:05,776::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 03:56:05,777::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:56:05,778::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:56:05,778::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:56:05,778::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:56:05,779::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 03:56:05,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:56:06,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:56:06,191::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:56:06,192::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 03:56:06,194::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 03:56:06,196::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 03:56:06,196::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 03:56:06,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:56:06,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:56:06,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:56:06,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00563213 s, 727 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:56:07,655::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:56:07,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142571 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:56:16,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:56:16,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154351 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:56:17,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:56:17,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149623 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14960::WARNING::2013-03-18 03:56:19,799::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:56:19,802::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:56:26,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:56:26,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784897 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:56:27,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:56:27,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092178 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:56:29,843::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:56:29,845::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:56:30,240::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:56:30,246::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:56:30,248::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:56:36,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:56:36,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132271 s, 310 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:56:37,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:56:37,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142409 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:56:46,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:56:46,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158796 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:56:47,826::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 03:56:47,828::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 03:56:47,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:56:47,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00188545 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:56:47,865::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00188545 s, 1.1 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 03:56:47,865::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 03:56:47,866::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 03:56:47,867::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 03:56:47,868::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:56:47,869::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:56:47,869::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:56:47,870::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:56:47,870::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 03:56:47,871::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:56:48,287::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:56:48,291::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 03:56:48,291::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 03:56:48,294::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 03:56:48,295::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 03:56:48,295::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 03:56:48,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:56:48,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:56:48,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:56:48,793::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0262424 s, 156 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:56:57,000::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:56:57,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799353 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:56:58,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:56:58,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105593 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:57:07,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:57:07,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129905 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:57:08,837::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:57:08,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817345 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:57:17,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:57:17,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153945 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:57:18,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:57:18,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752514 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:57:27,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:57:27,302::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141281 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:57:28,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:57:29,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806436 s, 5.1 MB/s\n'; <rc> = 0
>Thread-14961::WARNING::2013-03-18 03:57:30,285::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:57:30,289::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:57:37,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:57:37,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000713776 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:57:39,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:57:39,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919715 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:57:40,291::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 03:57:40,292::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:57:40,293::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:57:40,729::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:57:40,736::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:57:40,737::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:57:47,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:57:47,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128229 s, 319 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:57:49,092::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:57:49,123::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135849 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:57:57,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:57:57,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113392 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:57:59,129::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:57:59,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143177 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:58:07,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:58:07,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130705 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:58:09,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:58:09,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156149 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:58:17,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:58:17,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000694526 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:58:19,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:58:19,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078384 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:58:27,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:58:27,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000745048 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:58:29,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:58:29,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843572 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:58:37,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:58:37,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753991 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:58:39,308::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:58:39,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126603 s, 3.2 MB/s\n'; <rc> = 0
>Thread-14962::WARNING::2013-03-18 03:58:40,766::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:58:40,771::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:58:47,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:58:47,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760111 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:58:49,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:58:49,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129105 s, 3.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 03:58:50,778::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 03:58:50,780::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 03:58:51,241::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 03:58:51,248::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 03:58:51,249::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 03:58:57,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:58:58,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184123 s, 222 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:58:59,439::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:58:59,481::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130768 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:59:08,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:59:08,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138738 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:59:09,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:59:09,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142796 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:59:18,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:59:18,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141459 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:59:19,530::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:59:19,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162507 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:59:28,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:59:28,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823705 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:59:29,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:59:29,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787259 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:59:38,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:59:38,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00815098 s, 503 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:59:39,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:59:39,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134415 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 03:59:48,396::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:59:48,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119369 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:59:49,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:59:49,709::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109507 s, 3.7 MB/s\n'; <rc> = 0
>Thread-14963::WARNING::2013-03-18 03:59:51,276::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 03:59:51,279::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 03:59:58,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 03:59:58,529::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804875 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 03:59:59,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 03:59:59,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168638 s, 2.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:00:01,282::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:00:01,283::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:00:02,019::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:00:02,026::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:00:02,028::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:00:08,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:00:08,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160047 s, 256 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:00:09,755::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:00:09,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917702 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:00:18,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:00:18,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761581 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:00:19,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:00:19,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146678 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:00:28,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:00:28,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0230583 s, 178 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:00:29,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:00:29,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144458 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:00:38,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:00:38,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823552 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:00:39,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:00:39,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804462 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:00:48,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:00:48,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139492 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:00:49,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:00:49,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124705 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:00:59,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:00:59,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000929526 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:00:59,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:01:00,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00274133 s, 1.5 MB/s\n'; <rc> = 0
>Thread-14964::WARNING::2013-03-18 04:01:02,061::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:01:02,066::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 04:01:09,080::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 04:01:09,082::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 04:01:09,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:01:09,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00085087 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:01:09,111::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00085087 s, 2.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 04:01:09,111::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 04:01:09,111::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 04:01:09,113::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 04:01:09,114::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:01:09,115::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:01:09,115::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:01:09,115::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:01:09,116::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:01:09,117::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:01:09,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:01:09,579::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:01:09,580::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 04:01:09,583::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 04:01:09,584::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 04:01:09,585::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 04:01:09,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:01:09,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:01:09,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:01:10,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00982103 s, 417 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:01:10,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:01:10,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154104 s, 266 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:01:12,069::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:01:12,070::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:01:12,539::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:01:12,545::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:01:12,547::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:01:20,108::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:01:20,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:01:20,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00275262 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:01:20,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148961 s, 275 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:01:30,200::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:01:30,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133054 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:01:30,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:01:30,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171069 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:01:40,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:01:40,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00202912 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:01:40,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:01:40,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00935948 s, 438 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:01:50,329::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 04:01:50,331::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 04:01:50,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:01:50,364::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00127606 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:01:50,365::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00127606 s, 1.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 04:01:50,365::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 04:01:50,366::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 04:01:50,367::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 04:01:50,368::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:01:50,369::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:01:50,369::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:01:50,370::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:01:50,370::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:01:50,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:01:50,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:01:50,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142993 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:01:50,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:01:50,819::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:01:50,820::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 04:01:50,822::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 04:01:50,823::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 04:01:50,824::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 04:01:50,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:01:51,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:01:51,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:01:51,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00634159 s, 646 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:02:00,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:02:00,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115676 s, 354 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:02:01,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:02:01,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00276958 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:02:10,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:02:10,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146539 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:02:11,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:02:11,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00366718 s, 1.1 MB/s\n'; <rc> = 0
>Thread-14965::WARNING::2013-03-18 04:02:12,587::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:02:12,593::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 04:02:20,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:02:20,744::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156935 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:02:21,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:02:21,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131967 s, 3.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:02:22,596::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:02:22,598::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:02:23,037::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:02:23,043::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:02:23,045::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:02:30,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:02:30,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180334 s, 227 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:02:31,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:02:31,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152599 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:02:40,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:02:40,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841019 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:02:41,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:02:41,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000739298 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:02:50,944::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:02:50,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887761 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:02:51,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:02:51,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00183027 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:03:01,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:03:01,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00190547 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:03:01,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:03:01,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138492 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:03:11,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:03:11,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014485 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:03:11,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:03:11,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000732239 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:03:21,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:03:21,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000971038 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:03:21,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:03:21,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150825 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14966::WARNING::2013-03-18 04:03:23,070::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:03:23,074::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 04:03:31,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:03:31,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902876 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:03:31,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:03:31,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000954858 s, 4.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:03:33,077::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 04:03:33,078::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:03:33,080::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:03:33,564::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:03:33,570::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:03:33,572::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:03:41,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:03:41,409::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151165 s, 271 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:03:41,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:03:41,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142157 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:03:51,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:03:51,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833589 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:03:51,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:03:51,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000970472 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:04:01,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:04:01,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00250114 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:04:01,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:04:02,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014857 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:04:11,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:04:11,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823189 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:04:12,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:04:12,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00684447 s, 598 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:04:21,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:04:21,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868899 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:04:22,070::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:04:22,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134751 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:04:31,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:04:31,805::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000939512 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:04:32,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:04:32,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104326 s, 393 kB/s\n'; <rc> = 0
>Thread-14967::WARNING::2013-03-18 04:04:33,596::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:04:33,599::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 04:04:41,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:04:41,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138806 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:04:42,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:04:42,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836107 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:04:43,605::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:04:43,606::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:04:44,123::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:04:44,130::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:04:44,131::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:04:51,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:04:51,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183103 s, 224 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:04:52,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:04:52,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092254 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:05:02,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:05:02,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747803 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:05:02,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:05:02,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899767 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:05:12,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:05:12,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882601 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:05:12,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:05:12,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00223088 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:05:22,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:05:22,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153128 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:05:22,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:05:22,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814998 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:05:32,221::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:05:32,266::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164985 s, 248 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:05:32,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:05:32,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130052 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:05:42,312::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:05:42,350::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136563 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:05:42,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:05:42,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000745494 s, 5.5 MB/s\n'; <rc> = 0
>Thread-14968::WARNING::2013-03-18 04:05:44,157::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:05:44,161::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 04:05:52,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:05:52,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877342 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:05:52,465::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:05:52,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000964148 s, 4.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:05:54,163::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:05:54,165::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:05:54,609::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:05:54,616::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:05:54,617::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:06:02,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:06:02,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:06:02,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119225 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:06:02,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186852 s, 219 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:06:12,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:06:12,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00280876 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:06:12,682::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 04:06:12,683::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 04:06:12,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:06:12,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000762332 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:06:12,710::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000762332 s, 2.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 04:06:12,710::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 04:06:12,711::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 04:06:12,712::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 04:06:12,713::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:06:12,714::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:06:12,715::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:06:12,715::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:06:12,715::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:06:12,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:06:13,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:06:13,142::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:06:13,144::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 04:06:13,148::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 04:06:13,149::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 04:06:13,149::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 04:06:13,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:06:13,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:06:13,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:06:13,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145205 s, 282 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:06:22,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:06:22,700::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886317 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:06:23,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:06:23,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780614 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:06:32,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:06:32,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138672 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:06:33,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:06:33,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136025 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:06:42,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:06:42,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128007 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:06:43,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:06:43,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743146 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:06:52,804::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 04:06:52,809::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 04:06:52,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:06:52,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000767397 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:06:52,848::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000767397 s, 2.7 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 04:06:52,848::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 04:06:52,848::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 04:06:52,850::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 04:06:52,851::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:06:52,852::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:06:52,853::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:06:52,853::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:06:52,853::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:06:52,854::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:06:53,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:06:53,341::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:06:53,342::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 04:06:53,345::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 04:06:53,348::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 04:06:53,348::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 04:06:53,350::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:06:53,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:06:53,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184322 s, 222 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:06:54,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:06:54,108::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:06:54,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00713571 s, 574 kB/s\n'; <rc> = 0
>Thread-14969::WARNING::2013-03-18 04:06:54,652::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:06:54,656::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:07:04,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:07:04,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:07:04,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000697159 s, 5.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:07:04,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0273201 s, 150 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:07:04,658::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:07:04,660::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:07:05,103::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:07:05,110::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:07:05,111::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:07:14,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:07:14,404::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144778 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:07:14,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:07:14,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793095 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:07:24,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:07:24,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164162 s, 250 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:07:24,492::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:07:24,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815755 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:07:34,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:07:34,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00238049 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:07:34,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:07:34,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00270177 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:07:44,530::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:07:44,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793301 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:07:44,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:07:44,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.004968 s, 824 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:07:54,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:07:54,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107161 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:07:54,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:07:54,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00223724 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:08:04,605::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:08:04,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153529 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:08:04,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:08:04,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000692078 s, 5.9 MB/s\n'; <rc> = 0
>Thread-14970::WARNING::2013-03-18 04:08:05,141::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:08:05,147::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:08:14,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:08:14,819::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00574188 s, 713 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:08:15,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:08:15,153::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:08:15,157::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:08:15,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0262392 s, 156 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:08:15,644::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:08:15,649::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:08:15,650::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:08:24,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:08:24,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00992876 s, 413 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:08:25,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:08:25,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156602 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:08:34,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:08:34,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100134 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:08:35,341::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:08:35,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017384 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:08:44,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:08:44,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819029 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:08:45,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:08:45,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158077 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:08:54,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:08:54,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843204 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:08:55,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:08:55,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748584 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:09:04,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:09:05,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001454 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:09:05,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:09:05,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001358 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:09:15,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:09:15,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00287287 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:09:15,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-14971::WARNING::2013-03-18 04:09:15,686::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:09:15,720::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 04:09:15,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149037 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:09:25,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:09:25,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0081237 s, 504 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:09:25,723::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 04:09:25,725::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:09:25,726::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:09:25,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:09:25,819::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00724869 s, 565 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:09:26,214::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:09:26,219::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:09:26,221::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:09:35,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:09:35,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781868 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:09:35,863::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:09:35,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102243 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:09:45,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:09:45,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00210451 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:09:45,944::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:09:45,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000993814 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:09:55,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:09:55,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773026 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:09:56,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:09:56,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907459 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:10:05,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:10:05,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014111 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:10:06,079::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:10:06,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139169 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:10:15,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:10:15,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139328 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:10:16,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:10:16,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147153 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:10:25,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:10:25,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136479 s, 3.0 MB/s\n'; <rc> = 0
>Thread-14972::WARNING::2013-03-18 04:10:26,244::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-20::DEBUG::2013-03-18 04:10:26,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-21::ERROR::2013-03-18 04:10:26,254::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 04:10:26,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176458 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:10:35,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:10:35,649::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107848 s, 380 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:10:36,307::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:10:36,309::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:10:36,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:10:36,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152184 s, 269 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:10:36,747::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:10:36,752::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:10:36,754::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:10:45,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:10:45,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016743 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:10:46,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:10:46,478::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00095006 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:10:55,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:10:55,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147048 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:10:56,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:10:56,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117764 s, 348 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:11:05,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:11:05,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147337 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:11:06,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:11:06,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153556 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:11:15,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:11:15,843::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174515 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:11:16,693::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 04:11:16,695::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 04:11:16,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:11:16,744::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0093949 s, 218 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:11:16,745::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0093949 s, 218 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 04:11:16,746::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 04:11:16,746::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 04:11:16,747::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 04:11:16,748::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:11:16,749::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:11:16,750::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:11:16,750::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:11:16,750::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:11:16,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:11:17,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:11:17,185::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:11:17,186::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 04:11:17,189::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 04:11:17,190::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 04:11:17,190::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 04:11:17,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:11:17,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:11:17,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:11:17,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0327217 s, 125 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:11:25,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:11:25,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817851 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:11:27,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:11:27,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136201 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:11:35,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:11:36,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778501 s, 5.3 MB/s\n'; <rc> = 0
>Thread-14973::WARNING::2013-03-18 04:11:36,786::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:11:36,788::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 04:11:37,814::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:11:37,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000971436 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:11:46,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:11:46,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00927617 s, 442 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:11:46,791::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:11:46,792::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:11:47,237::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:11:47,243::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:11:47,245::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:11:47,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:11:47,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0284196 s, 144 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:11:56,170::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 04:11:56,172::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 04:11:56,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:11:56,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000984579 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:11:56,200::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000984579 s, 2.1 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 04:11:56,200::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 04:11:56,200::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 04:11:56,202::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 04:11:56,203::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:11:56,206::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:11:56,206::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:11:56,206::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:11:56,207::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:11:56,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:11:56,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:11:56,625::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:11:56,626::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 04:11:56,628::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 04:11:56,629::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 04:11:56,629::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 04:11:56,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:11:56,973::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:11:56,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:11:57,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00756268 s, 542 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:11:58,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:11:58,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016272 s, 252 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:12:07,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:12:07,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156172 s, 262 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:12:08,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:12:08,326::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00950913 s, 431 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:12:17,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:12:17,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915095 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:12:18,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:12:18,404::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127348 s, 322 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:12:27,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:12:27,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000966009 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:12:28,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:12:28,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925608 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:12:37,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:12:37,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767431 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:12:38,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:12:38,561::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00705036 s, 581 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:12:47,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:12:47,236::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151365 s, 2.7 MB/s\n'; <rc> = 0
>Thread-14974::WARNING::2013-03-18 04:12:47,271::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:12:47,275::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 04:12:48,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:12:48,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011478 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:12:57,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:12:57,279::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:12:57,281::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:12:57,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000990704 s, 4.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:12:57,725::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:12:57,731::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:12:57,732::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:12:58,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:12:58,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00939677 s, 436 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:13:07,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:13:07,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871531 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:13:08,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:13:08,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139108 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:13:17,341::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:13:17,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832835 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:13:18,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:13:18,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825186 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:13:27,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:13:27,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169437 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:13:28,938::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:13:28,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000999217 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:13:37,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:13:37,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864287 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:13:39,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:13:39,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769035 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:13:47,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:13:47,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155773 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:13:49,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:13:49,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164664 s, 249 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:13:57,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:13:57,575::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103219 s, 4.0 MB/s\n'; <rc> = 0
>Thread-14975::WARNING::2013-03-18 04:13:57,756::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:13:57,762::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 04:13:59,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:13:59,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869655 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:14:07,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:14:07,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170264 s, 241 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:14:07,765::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:14:07,766::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:14:08,237::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:14:08,244::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:14:08,245::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:14:09,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:14:09,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0196455 s, 208 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:14:17,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:14:17,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00274356 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:14:19,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:14:19,392::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165824 s, 247 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:14:27,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:14:27,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766552 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:14:29,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:14:29,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858272 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:14:37,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:14:37,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141654 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:14:39,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:14:39,542::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013895 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:14:47,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:14:47,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012302 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:14:49,583::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:14:49,614::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766753 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:14:57,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:14:57,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013892 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:14:59,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:14:59,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00201006 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:15:07,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:15:07,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909303 s, 4.5 MB/s\n'; <rc> = 0
>Thread-14976::WARNING::2013-03-18 04:15:08,279::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:15:08,283::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 04:15:09,741::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:15:09,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885859 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:15:17,941::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:15:18,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139178 s, 294 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:15:18,285::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 04:15:18,286::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:15:18,288::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:15:18,757::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:15:18,763::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:15:18,765::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:15:19,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:15:19,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136703 s, 300 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:15:28,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:15:28,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136706 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:15:29,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:15:29,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814462 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:15:38,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:15:38,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153257 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:15:39,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:15:39,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0114108 s, 359 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:15:48,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:15:48,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901211 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:15:50,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:15:50,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00606328 s, 676 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:15:58,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:15:58,197::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136074 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:16:00,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:16:00,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089069 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:16:08,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:16:08,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157813 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:16:10,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:16:10,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013684 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:16:18,257::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:16:18,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158975 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14977::WARNING::2013-03-18 04:16:18,795::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:16:18,799::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 04:16:20,269::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 04:16:20,270::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 04:16:20,271::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:16:20,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000787086 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:16:20,297::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000787086 s, 2.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 04:16:20,297::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 04:16:20,297::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 04:16:20,299::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 04:16:20,300::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:16:20,301::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:16:20,301::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:16:20,301::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:16:20,302::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:16:20,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:16:20,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:16:20,731::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:16:20,732::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 04:16:20,735::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 04:16:20,736::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 04:16:20,736::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 04:16:20,738::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:16:21,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:16:21,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:16:21,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171884 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:16:28,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:16:28,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.019508 s, 210 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:16:28,802::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:16:28,804::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:16:29,264::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:16:29,271::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:16:29,272::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:16:31,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:16:31,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134927 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:16:38,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:16:38,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780971 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:16:41,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:16:41,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133459 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:16:48,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:16:48,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131978 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:16:51,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:16:51,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135108 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:16:58,499::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 04:16:58,500::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 04:16:58,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:16:58,548::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00152948 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:16:58,549::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00152948 s, 1.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 04:16:58,550::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 04:16:58,550::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 04:16:58,552::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 04:16:58,553::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:16:58,553::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:16:58,554::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:16:58,554::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:16:58,555::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:16:58,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:16:58,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:16:59,003::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:16:59,004::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 04:16:59,006::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 04:16:59,007::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 04:16:59,008::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 04:16:59,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:16:59,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:16:59,372::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:16:59,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134355 s, 305 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:17:01,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:17:01,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124051 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:17:09,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:17:09,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013889 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:17:11,522::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:17:11,561::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014325 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:17:19,481::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:17:19,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778782 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:17:21,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:17:21,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00283898 s, 1.4 MB/s\n'; <rc> = 0
>Thread-14978::WARNING::2013-03-18 04:17:29,299::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:17:29,302::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:17:29,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:17:29,552::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787822 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:17:31,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:17:31,734::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884325 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:17:39,304::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:17:39,306::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:17:39,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:17:39,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129853 s, 315 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:17:39,771::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:17:39,777::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:17:39,778::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:17:41,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:17:41,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014712 s, 278 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:17:49,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:17:49,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142747 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:17:51,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:17:51,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795115 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:17:59,685::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:17:59,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810776 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:18:01,938::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:18:01,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157873 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:18:09,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:18:09,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137203 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:18:12,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:18:12,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000889614 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:18:19,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:18:19,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133689 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:18:22,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:18:22,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149386 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:18:29,808::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:18:29,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143774 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:18:32,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:18:32,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000910268 s, 4.5 MB/s\n'; <rc> = 0
>Thread-14979::WARNING::2013-03-18 04:18:39,807::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:18:39,811::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:18:39,878::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:18:39,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860356 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:18:42,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:18:42,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015322 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:18:49,813::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:18:49,814::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:18:49,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:18:50,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169095 s, 242 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:18:50,279::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:18:50,285::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:18:50,287::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:18:52,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:18:52,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.020497 s, 200 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:19:00,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:19:00,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143083 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:19:02,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:19:02,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134321 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:19:10,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:19:10,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801765 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:19:12,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:19:12,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859843 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:19:20,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:19:20,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151827 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:19:22,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:19:22,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879488 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:19:30,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:19:30,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012396 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:19:32,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:19:32,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874609 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:19:40,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:19:40,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000742929 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:19:42,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:19:42,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790691 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:19:50,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-14980::WARNING::2013-03-18 04:19:50,339::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-22::DEBUG::2013-03-18 04:19:50,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00205135 s, 2.0 MB/s\n'; <rc> = 0
>Thread-21::ERROR::2013-03-18 04:19:50,356::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 04:19:52,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:19:52,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809476 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:20:00,359::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:20:00,360::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:20:00,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:20:00,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0207984 s, 197 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:20:00,836::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:20:00,842::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:20:00,843::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:20:02,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:20:02,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00645398 s, 635 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:20:10,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:20:10,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864958 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:20:12,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:20:12,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120489 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:20:20,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:20:20,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134071 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:20:23,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:20:23,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000706203 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:20:30,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:20:30,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130645 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:20:33,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:20:33,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832956 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:20:40,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:20:40,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000745932 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:20:43,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:20:43,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164845 s, 248 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:20:50,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:20:50,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127952 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:20:53,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:20:53,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00193224 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:21:00,687::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:21:00,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798839 s, 5.1 MB/s\n'; <rc> = 0
>Thread-14981::WARNING::2013-03-18 04:21:00,869::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:21:00,873::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 04:21:03,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:21:03,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000921366 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:21:10,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:21:10,820::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016512 s, 248 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:21:10,875::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 04:21:10,876::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:21:10,878::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:21:11,295::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:21:11,302::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:21:11,304::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:21:13,429::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:21:13,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152871 s, 268 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:21:20,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:21:20,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00280943 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:21:23,511::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 04:21:23,513::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 04:21:23,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:21:23,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000688243 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:21:23,542::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000688243 s, 3.0 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 04:21:23,542::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 04:21:23,542::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 04:21:23,544::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 04:21:23,545::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:21:23,546::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:21:23,546::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:21:23,546::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:21:23,547::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:21:23,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:21:24,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:21:24,020::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:21:24,021::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 04:21:24,024::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 04:21:24,025::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 04:21:24,025::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 04:21:24,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:21:24,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:21:24,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:21:24,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170097 s, 241 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:21:30,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:21:30,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000982143 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:21:34,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:21:34,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916057 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:21:40,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:21:40,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134126 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:21:44,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:21:44,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870545 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:21:50,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:21:50,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887086 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:21:54,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:21:54,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138663 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:22:00,985::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 04:22:00,987::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 04:22:00,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:22:01,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00208252 s, 983 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:22:01,030::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00208252 s, 983 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 04:22:01,031::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 04:22:01,032::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 04:22:01,033::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 04:22:01,034::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:22:01,035::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:22:01,035::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:22:01,035::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:22:01,036::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:22:01,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:22:01,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:22:01,477::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:22:01,478::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 04:22:01,485::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 04:22:01,487::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 04:22:01,487::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 04:22:01,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:22:01,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:22:01,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:22:01,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171255 s, 239 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:22:04,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:22:04,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169321 s, 242 kB/s\n'; <rc> = 0
>Thread-14982::WARNING::2013-03-18 04:22:11,331::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:22:11,335::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:22:11,959::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:22:11,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833068 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:22:14,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:22:15,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869839 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:22:21,338::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:22:21,339::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:22:21,752::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:22:21,758::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:22:21,760::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:22:21,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:22:22,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161794 s, 253 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:22:25,070::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:22:25,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135286 s, 303 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:22:32,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:22:32,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00194677 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:22:35,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:22:35,175::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749127 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:22:42,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:22:42,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010956 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:22:45,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:22:45,266::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00543563 s, 754 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:22:52,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:22:52,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130636 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:22:55,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:22:55,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144752 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:23:02,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:23:02,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132635 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:23:05,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:23:05,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825515 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:23:12,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:23:12,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00235919 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:23:15,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:23:15,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000678045 s, 6.0 MB/s\n'; <rc> = 0
>Thread-14983::WARNING::2013-03-18 04:23:21,785::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:23:21,788::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:23:22,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:23:22,297::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822983 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:23:25,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:23:25,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163398 s, 2.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:23:31,790::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:23:31,792::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:23:32,243::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:23:32,249::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:23:32,251::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:23:32,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:23:32,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153205 s, 267 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:23:35,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:23:35,657::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00687268 s, 596 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:23:42,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:23:42,415::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010438 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:23:45,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:23:45,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084765 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:23:52,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:23:52,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827555 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:23:55,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:23:55,793::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760407 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:24:02,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:24:02,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839667 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:24:05,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:24:05,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014098 s, 291 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:24:12,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:24:12,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166835 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:24:15,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:24:15,958::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148138 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:24:22,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:24:22,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117101 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:24:26,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:24:26,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138881 s, 2.9 MB/s\n'; <rc> = 0
>Thread-14984::WARNING::2013-03-18 04:24:32,278::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:24:32,285::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:24:32,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:24:32,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135192 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:24:36,091::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:24:36,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911142 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:24:42,287::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:24:42,288::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:24:42,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:24:42,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.020224 s, 203 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:24:42,711::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:24:42,717::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:24:42,718::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:24:46,156::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:24:46,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156849 s, 261 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:24:52,685::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:24:52,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120261 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:24:56,243::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:24:56,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176028 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:25:02,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:25:02,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000990771 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:25:06,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:25:06,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00605113 s, 677 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:25:12,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:25:12,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00185913 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:25:16,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:25:16,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837767 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:25:22,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:25:22,907::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150367 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:25:26,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:25:26,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166751 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:25:32,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:25:32,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139359 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:25:36,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:25:36,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159135 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14985::WARNING::2013-03-18 04:25:42,744::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:25:42,747::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:25:42,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:25:43,010::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175181 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:25:46,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:25:46,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016547 s, 248 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:25:52,749::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:25:52,751::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:25:53,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:25:53,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132387 s, 309 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:25:53,195::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:25:53,201::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:25:53,202::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:25:56,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:25:56,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170884 s, 240 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:26:03,080::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:26:03,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085568 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:26:06,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:26:06,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091874 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:26:13,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:26:13,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865119 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:26:16,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:26:16,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782336 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:26:23,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:26:23,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771709 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:26:27,005::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 04:26:27,006::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 04:26:27,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:26:27,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00134726 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:26:27,040::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00134726 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 04:26:27,041::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 04:26:27,041::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 04:26:27,042::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 04:26:27,043::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:26:27,044::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:26:27,045::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:26:27,045::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:26:27,045::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:26:27,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:26:27,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:26:27,463::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:26:27,464::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 04:26:27,466::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 04:26:27,467::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 04:26:27,468::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 04:26:27,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:26:27,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:26:27,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:26:27,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00665398 s, 616 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:26:33,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:26:33,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826517 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:26:37,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:26:37,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000729767 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:26:43,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:26:43,266::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175475 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:26:47,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:26:48,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015625 s, 2.6 MB/s\n'; <rc> = 0
>Thread-14986::WARNING::2013-03-18 04:26:53,228::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:26:53,232::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:26:53,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:26:53,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870478 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:26:58,049::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:26:58,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802207 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:27:03,234::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 04:27:03,235::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:27:03,236::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:27:03,311::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 04:27:03,314::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 04:27:03,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:27:03,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0124947 s, 164 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:27:03,378::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0124947 s, 164 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 04:27:03,378::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 04:27:03,379::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 04:27:03,381::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 04:27:03,382::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm reload operation' is holding the operation mutex, waiting...
>Thread-21::DEBUG::2013-03-18 04:27:03,683::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:27:03,691::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:27:03,692::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:27:03,700::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:27:03,701::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:27:03,702::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:27:03,702::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:27:03,702::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:27:03,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:27:04,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:27:04,101::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:27:04,102::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 04:27:04,104::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 04:27:04,106::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 04:27:04,106::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 04:27:04,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:27:04,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:27:04,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:27:04,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00774161 s, 529 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:27:08,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:27:08,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170529 s, 240 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:27:14,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:27:14,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824072 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:27:18,230::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:27:18,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135059 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:27:24,532::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:27:24,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130101 s, 315 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:27:28,312::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:27:28,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014259 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:27:34,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:27:34,622::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015227 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:27:38,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:27:38,413::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797325 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:27:44,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:27:44,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784998 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:27:48,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:27:48,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000935856 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:27:54,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:27:54,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000996888 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:27:58,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:27:58,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000947485 s, 4.3 MB/s\n'; <rc> = 0
>Thread-14987::WARNING::2013-03-18 04:28:03,763::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:28:03,769::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:28:04,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:28:04,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783788 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:28:08,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:28:08,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883992 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:28:13,783::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:28:13,784::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:28:14,234::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:28:14,241::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:28:14,243::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:28:14,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:28:14,818::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161159 s, 254 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:28:18,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:28:18,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126986 s, 323 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:28:24,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:28:24,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103911 s, 394 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:28:28,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:28:28,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00099245 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:28:34,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:28:34,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00201758 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:28:38,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:28:38,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000672296 s, 6.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:28:44,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:28:44,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013771 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:28:48,898::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:28:48,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133474 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:28:54,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:28:54,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135972 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:28:58,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:28:59,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156914 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:29:05,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:29:05,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141839 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:29:09,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:29:09,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109421 s, 3.7 MB/s\n'; <rc> = 0
>Thread-14988::WARNING::2013-03-18 04:29:14,281::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:29:14,285::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:29:15,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:29:15,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836366 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:29:19,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:29:19,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148928 s, 275 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:29:24,287::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:29:24,288::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:29:24,747::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:29:24,753::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:29:24,755::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:29:25,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:29:25,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179113 s, 229 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:29:29,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:29:29,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135836 s, 302 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:29:35,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:29:35,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936123 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:29:39,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:29:39,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850321 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:29:45,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:29:45,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904397 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:29:49,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:29:49,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181537 s, 226 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:29:55,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:29:55,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786746 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:29:59,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:29:59,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131678 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:30:05,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:30:05,299::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139989 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:30:09,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:30:09,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077693 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:30:15,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:30:15,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00630103 s, 650 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:30:19,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:30:19,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790092 s, 5.2 MB/s\n'; <rc> = 0
>Thread-14989::WARNING::2013-03-18 04:30:24,779::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:30:24,783::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:30:25,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:30:25,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786863 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:30:29,851::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:30:29,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077958 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:30:34,789::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:30:34,790::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:30:35,257::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:30:35,263::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:30:35,264::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:30:35,389::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:30:35,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152344 s, 269 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:30:39,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:30:39,968::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133661 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:30:45,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:30:45,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164566 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:30:50,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:30:50,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000917772 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:30:55,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:30:55,529::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861032 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:31:00,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:31:00,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139541 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:31:05,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:31:05,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899032 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:31:10,174::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:31:10,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159466 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:31:15,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:31:15,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001559 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:31:20,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:31:20,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131365 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:31:25,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:31:25,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135366 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:31:30,328::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 04:31:30,330::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 04:31:30,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:31:30,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00075044 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:31:30,361::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00075044 s, 2.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 04:31:30,362::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 04:31:30,362::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 04:31:30,367::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 04:31:30,368::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:31:30,369::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:31:30,369::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:31:30,369::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:31:30,379::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:31:30,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:31:30,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:31:30,834::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:31:30,835::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 04:31:30,837::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 04:31:30,839::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 04:31:30,839::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 04:31:30,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:31:31,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:31:31,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:31:31,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00665758 s, 615 kB/s\n'; <rc> = 0
>Thread-14990::WARNING::2013-03-18 04:31:35,291::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:31:35,294::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:31:35,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:31:35,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147362 s, 278 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:31:41,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:31:41,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083342 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:31:45,296::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:31:45,298::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:31:45,704::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:31:45,711::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:31:45,712::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:31:45,717::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:31:45,784::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138091 s, 297 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:31:51,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:31:51,398::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790464 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:31:55,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:31:55,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174635 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:32:01,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:32:01,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772009 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:32:05,837::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 04:32:05,838::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 04:32:05,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:32:05,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000772561 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:32:05,867::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000772561 s, 2.7 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 04:32:05,867::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 04:32:05,867::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 04:32:05,870::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 04:32:05,871::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:32:05,871::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:32:05,872::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:32:05,872::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:32:05,872::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:32:05,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:32:06,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:32:06,313::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:32:06,314::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 04:32:06,316::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 04:32:06,318::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 04:32:06,318::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 04:32:06,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:32:06,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:32:06,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:32:06,739::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140004 s, 293 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:32:11,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:32:11,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125826 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:32:16,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:32:16,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00269802 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:32:21,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:32:21,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815846 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:32:26,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:32:26,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129097 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:32:31,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:32:31,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779921 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:32:36,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:32:36,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141758 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:32:41,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:32:41,766::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146118 s, 2.8 MB/s\n'; <rc> = 0
>Thread-14991::WARNING::2013-03-18 04:32:45,761::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:32:45,765::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:32:46,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:32:46,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00393547 s, 1.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:32:51,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:32:51,843::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869182 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:32:55,766::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 04:32:55,767::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:32:55,768::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:32:56,193::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:32:56,200::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:32:56,201::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:32:56,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:32:56,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00581433 s, 704 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:33:01,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:33:01,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132557 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:33:06,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:33:07,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144907 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:33:11,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:33:12,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757135 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:33:17,019::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:33:17,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000953631 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:33:22,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:33:22,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144158 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:33:27,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:33:27,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769329 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:33:32,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:33:32,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846797 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:33:37,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:33:37,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142297 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:33:42,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:33:42,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142209 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:33:47,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:33:47,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791205 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:33:52,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:33:52,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839585 s, 4.9 MB/s\n'; <rc> = 0
>Thread-14992::WARNING::2013-03-18 04:33:56,226::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:33:56,229::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:33:57,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:33:57,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000674539 s, 6.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:34:02,385::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:34:02,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101784 s, 4.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:34:06,231::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:34:06,233::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:34:06,689::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:34:06,695::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:34:06,697::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:34:07,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:34:07,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119116 s, 344 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:34:12,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:34:12,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164522 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:34:17,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:34:17,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851896 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:34:22,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:34:22,561::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913971 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:34:27,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:34:27,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120316 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:34:32,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:34:32,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805545 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:34:37,376::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:34:37,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133615 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:34:42,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:34:42,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766253 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:34:47,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:34:47,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814732 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:34:52,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:34:52,781::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101913 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:34:57,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:34:57,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000922495 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:35:02,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:35:02,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00243334 s, 1.7 MB/s\n'; <rc> = 0
>Thread-14993::WARNING::2013-03-18 04:35:06,727::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:35:06,731::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:35:07,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:35:07,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806936 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:35:12,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:35:12,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816138 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:35:16,733::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:35:16,734::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:35:17,158::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:35:17,165::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:35:17,166::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:35:17,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:35:17,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135378 s, 303 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:35:22,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:35:23,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799781 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:35:27,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:35:27,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076234 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:35:33,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:35:33,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907921 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:35:37,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:35:37,681::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140953 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:35:43,178::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:35:43,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00391537 s, 1.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:35:47,686::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:35:47,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853118 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:35:53,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:35:53,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847892 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:35:57,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:35:57,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773543 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:36:03,427::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:36:03,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871817 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:36:07,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:36:07,805::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144545 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:36:13,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:36:13,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115657 s, 3.5 MB/s\n'; <rc> = 0
>Thread-14994::WARNING::2013-03-18 04:36:17,194::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:36:17,198::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:36:17,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:36:17,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810734 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:36:23,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:36:23,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897413 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:36:27,200::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:36:27,201::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:36:27,673::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:36:27,680::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:36:27,681::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:36:27,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:36:27,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177997 s, 230 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:36:33,664::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 04:36:33,666::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 04:36:33,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:36:33,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000776663 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:36:33,696::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000776663 s, 2.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 04:36:33,697::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 04:36:33,697::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 04:36:33,699::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 04:36:33,700::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:36:33,701::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:36:33,701::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:36:33,701::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:36:33,702::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:36:33,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:36:34,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:36:34,136::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:36:34,137::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 04:36:34,139::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 04:36:34,140::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 04:36:34,141::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 04:36:34,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:36:34,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:36:34,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:36:34,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00890812 s, 460 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:36:37,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:36:37,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188052 s, 218 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:36:44,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:36:44,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757557 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:36:47,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:36:47,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134568 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:36:54,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:36:54,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000734867 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:36:58,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:36:58,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139483 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:37:04,708::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:37:04,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150785 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:37:08,052::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 04:37:08,053::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 04:37:08,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:37:08,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00121244 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:37:08,102::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00121244 s, 1.7 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 04:37:08,102::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 04:37:08,103::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 04:37:08,104::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 04:37:08,105::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:37:08,106::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:37:08,106::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:37:08,106::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:37:08,107::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:37:08,108::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:37:08,539::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:37:08,543::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:37:08,544::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 04:37:08,547::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 04:37:08,548::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 04:37:08,549::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 04:37:08,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:37:08,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:37:08,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:37:09,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0352302 s, 116 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:37:14,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:37:14,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813352 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:37:19,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:37:19,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127555 s, 321 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:37:24,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:37:24,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00090951 s, 4.5 MB/s\n'; <rc> = 0
>Thread-14995::WARNING::2013-03-18 04:37:27,714::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:37:27,717::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:37:29,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:37:29,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874982 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:37:34,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:37:34,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141221 s, 2.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:37:37,719::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:37:37,721::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:37:38,122::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:37:38,129::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:37:38,130::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:37:39,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:37:39,152::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107502 s, 381 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:37:45,021::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:37:45,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137529 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:37:49,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:37:49,191::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805454 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:37:55,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:37:55,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141884 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:37:59,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:37:59,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152491 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:38:05,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:38:05,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846099 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:38:09,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:38:09,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127896 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:38:15,260::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:38:15,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133716 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:38:19,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:38:19,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000980596 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:38:25,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:38:25,372::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116429 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:38:29,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:38:29,398::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135374 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:38:35,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:38:35,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797808 s, 5.1 MB/s\n'; <rc> = 0
>Thread-14996::WARNING::2013-03-18 04:38:38,157::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:38:38,161::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:38:39,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:38:39,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738774 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:38:45,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:38:45,529::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826977 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:38:48,162::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 04:38:48,163::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:38:48,164::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:38:48,599::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:38:48,605::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:38:48,607::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:38:49,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:38:49,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.012451 s, 329 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:38:55,568::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:38:55,593::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876898 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:38:59,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:38:59,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847642 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:39:05,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:39:05,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867852 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:39:09,522::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:39:09,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813366 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:39:15,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:39:15,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106611 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:39:19,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:39:19,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816486 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:39:25,794::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:39:25,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762215 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:39:29,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:39:29,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139545 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:39:35,861::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:39:35,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000993623 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:39:39,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:39:39,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126337 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:39:45,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:39:45,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101405 s, 4.0 MB/s\n'; <rc> = 0
>Thread-14997::WARNING::2013-03-18 04:39:48,642::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:39:48,647::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:39:49,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:39:49,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015068 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:39:56,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:39:56,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135854 s, 3.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:39:58,650::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:39:58,652::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:39:59,079::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:39:59,087::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:39:59,088::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:39:59,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:39:59,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178898 s, 229 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:40:06,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:40:06,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119852 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:40:09,778::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:40:09,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144953 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:40:16,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:40:16,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781381 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:40:19,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:40:19,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001917 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:40:26,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:40:26,309::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000734023 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:40:29,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:40:29,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148919 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:40:36,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:40:36,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149883 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:40:39,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:40:39,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125445 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:40:46,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:40:46,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147642 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:40:49,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:40:49,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137306 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:40:56,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:40:56,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797921 s, 5.1 MB/s\n'; <rc> = 0
>Thread-14998::WARNING::2013-03-18 04:40:59,117::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:40:59,121::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:40:59,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:41:00,017::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838312 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:41:06,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:41:06,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173132 s, 2.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:41:09,123::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:41:09,124::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:41:09,550::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:41:09,557::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:41:09,558::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:41:10,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:41:10,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186102 s, 220 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:41:16,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:41:16,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00096746 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:41:20,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:41:20,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081186 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:41:26,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:41:26,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808845 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:41:30,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:41:30,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000943828 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:41:36,802::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 04:41:36,804::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 04:41:36,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:41:36,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000732974 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:41:36,835::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000732974 s, 2.8 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 04:41:36,835::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 04:41:36,835::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 04:41:36,837::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 04:41:36,838::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:41:36,839::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:41:36,839::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:41:36,839::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:41:36,840::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:41:36,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:41:37,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:41:37,276::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:41:37,277::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 04:41:37,280::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 04:41:37,281::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 04:41:37,281::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 04:41:37,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:41:37,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:41:37,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:41:37,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122984 s, 333 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:41:40,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:41:40,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016733 s, 245 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:41:47,781::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:41:47,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140258 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:41:50,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:41:50,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838967 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:41:57,875::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:41:57,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779332 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:42:00,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:42:00,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822629 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:42:07,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:42:08,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738624 s, 5.5 MB/s\n'; <rc> = 0
>Thread-14999::WARNING::2013-03-18 04:42:09,587::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:42:09,593::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:42:10,292::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 04:42:10,294::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 04:42:10,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:42:10,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000739179 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:42:10,320::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000739179 s, 2.8 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 04:42:10,321::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 04:42:10,321::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 04:42:10,322::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 04:42:10,323::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:42:10,324::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:42:10,325::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:42:10,325::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:42:10,325::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:42:10,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:42:10,742::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:42:10,746::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:42:10,747::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 04:42:10,749::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 04:42:10,750::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 04:42:10,751::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 04:42:10,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:42:11,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:42:11,171::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:42:11,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116778 s, 351 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:42:18,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:42:18,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865675 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:42:19,595::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:42:19,597::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:42:20,040::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:42:20,047::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:42:20,048::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:42:21,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:42:21,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0045178 s, 907 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:42:28,524::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:42:28,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824764 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:42:31,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:42:31,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00373979 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:42:38,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:42:38,657::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817799 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:42:41,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:42:41,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00286415 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:42:48,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:42:48,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000709135 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:42:51,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:42:51,382::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101914 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:42:58,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:42:58,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000707414 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:43:01,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:43:01,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158601 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:43:08,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:43:08,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085952 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:43:11,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:43:11,478::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151644 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:43:18,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:43:18,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000731362 s, 5.6 MB/s\n'; <rc> = 0
>Thread-15000::WARNING::2013-03-18 04:43:20,075::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:43:20,079::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:43:21,483::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:43:21,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869922 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:43:29,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:43:29,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015912 s, 2.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:43:30,082::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:43:30,083::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:43:30,495::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:43:30,502::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:43:30,504::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:43:31,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:43:31,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140938 s, 291 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:43:39,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:43:39,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136099 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:43:41,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:43:41,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134282 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:43:49,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:43:49,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000728923 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:43:51,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:43:51,646::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804911 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:43:59,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:43:59,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810132 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:44:01,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:44:01,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00243221 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:44:09,730::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:44:09,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883732 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:44:11,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:44:11,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000989146 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:44:19,800::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:44:19,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142547 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:44:21,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:44:21,766::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178992 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:44:29,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:44:29,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013281 s, 3.1 MB/s\n'; <rc> = 0
>Thread-15001::WARNING::2013-03-18 04:44:30,530::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:44:30,534::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:44:31,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:44:31,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913259 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:44:40,178::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:44:40,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138148 s, 296 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:44:40,536::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 04:44:40,540::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:44:40,541::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:44:40,957::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:44:40,964::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:44:40,965::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:44:41,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:44:41,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117109 s, 350 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:44:50,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:44:50,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821219 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:44:51,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:44:51,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838942 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:45:00,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:45:00,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831328 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:45:01,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:45:01,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839793 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:45:10,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:45:10,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000697586 s, 5.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:45:11,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:45:11,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171352 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:45:20,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:45:20,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136084 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:45:21,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:45:22,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829777 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:45:30,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:45:30,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151337 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:45:32,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:45:32,046::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140002 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:45:40,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:45:40,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125731 s, 326 kB/s\n'; <rc> = 0
>Thread-15002::WARNING::2013-03-18 04:45:40,992::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:45:41,002::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:45:42,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:45:42,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008012 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:45:51,014::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:45:51,016::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:45:51,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:45:51,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01895 s, 216 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:45:51,458::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:45:51,464::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:45:51,465::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:45:52,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:45:52,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105959 s, 387 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:46:01,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:46:01,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146864 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:46:02,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:46:02,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000903454 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:46:11,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:46:11,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792731 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:46:12,171::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:46:12,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137573 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:46:21,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:46:21,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872082 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:46:22,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:46:22,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932426 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:46:31,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:46:31,431::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142987 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:46:32,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:46:32,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143985 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:46:41,473::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 04:46:41,474::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 04:46:41,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:46:41,504::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00087341 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:46:41,505::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00087341 s, 2.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 04:46:41,506::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 04:46:41,506::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 04:46:41,508::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 04:46:41,509::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:46:41,510::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:46:41,510::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:46:41,511::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:46:41,511::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:46:41,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:46:41,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:46:41,935::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:46:41,936::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 04:46:41,938::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 04:46:41,940::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 04:46:41,940::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 04:46:41,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:46:42,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:46:42,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:46:42,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:46:42,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00184265 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:46:42,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148158 s, 276 kB/s\n'; <rc> = 0
>Thread-15003::WARNING::2013-03-18 04:46:51,501::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:46:51,507::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:46:52,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:46:52,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747729 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:46:52,429::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:46:52,453::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784546 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:47:01,509::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:47:01,510::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:47:01,936::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:47:01,942::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:47:01,944::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:47:02,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:47:02,490::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180867 s, 226 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:47:02,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:47:02,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137454 s, 298 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:47:12,537::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 04:47:12,538::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 04:47:12,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:47:12,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0020761 s, 986 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:47:12,568::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0020761 s, 986 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 04:47:12,568::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 04:47:12,568::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 04:47:12,570::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 04:47:12,571::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:47:12,572::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:47:12,572::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:47:12,572::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:47:12,573::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:47:12,574::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:47:12,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:47:12,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153458 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:47:13,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:47:13,010::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:47:13,011::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 04:47:13,013::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 04:47:13,015::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 04:47:13,015::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 04:47:13,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:47:13,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:47:13,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:47:13,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0202681 s, 202 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:47:22,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:47:22,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852459 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:47:23,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:47:23,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143382 s, 286 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:47:32,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:47:32,791::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789544 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:47:33,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:47:33,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812886 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:47:42,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:47:42,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902404 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:47:43,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:47:43,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00072404 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:47:52,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:47:52,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168129 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:47:53,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:47:53,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00198146 s, 2.1 MB/s\n'; <rc> = 0
>Thread-15004::WARNING::2013-03-18 04:48:01,983::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:48:01,987::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 04:48:02,983::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:48:03,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851275 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:48:03,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:48:03,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132111 s, 3.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:48:11,990::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:48:11,991::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:48:12,406::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:48:12,413::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:48:12,414::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:48:13,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:48:13,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748981 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:48:13,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:48:13,813::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121844 s, 336 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:48:23,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:48:23,163::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131213 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:48:23,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:48:23,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00430719 s, 951 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:48:33,202::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:48:33,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129328 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:48:33,863::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:48:33,893::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169722 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:48:43,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:48:43,322::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000713519 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:48:43,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:48:43,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156708 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:48:53,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:48:53,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138445 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:48:53,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:48:53,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804424 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:49:03,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:49:03,491::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00264699 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:49:03,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:49:04,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102455 s, 400 kB/s\n'; <rc> = 0
>Thread-15005::WARNING::2013-03-18 04:49:12,445::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:49:12,449::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 04:49:13,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:49:13,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794009 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:49:14,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:49:14,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00310892 s, 1.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:49:22,451::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:49:22,452::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:49:22,870::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:49:22,876::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:49:22,878::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:49:23,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:49:23,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842027 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:49:24,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:49:24,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00770102 s, 532 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:49:33,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:49:33,712::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00195422 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:49:34,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:49:34,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142964 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:49:43,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:49:43,783::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760925 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:49:44,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:49:44,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102498 s, 400 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:49:53,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:49:53,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141175 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:49:54,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:49:54,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103352 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:50:03,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:50:03,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132761 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:50:04,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:50:04,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870383 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:50:13,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:50:14,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00255119 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:50:14,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:50:14,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135717 s, 3.0 MB/s\n'; <rc> = 0
>Thread-15006::WARNING::2013-03-18 04:50:22,904::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:50:22,909::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 04:50:24,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:50:24,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135096 s, 303 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:50:24,341::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:50:24,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760159 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:50:32,911::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 04:50:32,913::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:50:32,913::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:50:33,409::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:50:33,416::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:50:33,417::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:50:34,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:50:34,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135565 s, 302 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:50:34,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:50:34,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163086 s, 251 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:50:44,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:50:44,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106229 s, 386 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:50:44,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:50:44,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000984188 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:50:54,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:50:54,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923601 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:50:54,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:50:54,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00329803 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:51:04,412::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:51:04,450::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108937 s, 376 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:51:04,521::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:51:04,561::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147487 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:51:14,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:51:14,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919729 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:51:14,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:51:14,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111366 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:51:24,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:51:24,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:51:24,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010205 s, 401 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:51:24,734::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133313 s, 3.1 MB/s\n'; <rc> = 0
>Thread-15007::WARNING::2013-03-18 04:51:33,443::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:51:33,453::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:51:34,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:51:34,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792596 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:51:34,781::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:51:34,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765361 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:51:43,457::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:51:43,458::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:51:43,886::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:51:43,893::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:51:43,894::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:51:44,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:51:44,856::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 04:51:44,875::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 04:51:44,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:51:44,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140667 s, 291 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:51:44,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0129612 s, 158 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:51:44,929::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0129612 s, 158 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 04:51:44,930::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 04:51:44,930::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 04:51:44,931::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 04:51:44,932::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:51:44,932::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:51:44,933::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:51:44,933::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:51:44,934::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:51:44,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:51:45,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:51:45,353::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:51:45,354::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 04:51:45,357::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 04:51:45,358::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 04:51:45,358::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 04:51:45,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:51:45,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:51:45,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:51:45,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0279829 s, 146 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:51:54,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:51:54,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134943 s, 304 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:51:55,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:51:55,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146206 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:52:04,993::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:52:05,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172872 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:52:05,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:52:05,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795333 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:52:15,025::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 04:52:15,026::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 04:52:15,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:52:15,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00134298 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:52:15,065::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00134298 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 04:52:15,066::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 04:52:15,066::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 04:52:15,067::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 04:52:15,068::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:52:15,069::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:52:15,070::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:52:15,070::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:52:15,070::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:52:15,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:52:15,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:52:15,486::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:52:15,487::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 04:52:15,489::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 04:52:15,490::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 04:52:15,491::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 04:52:15,492::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:52:15,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:52:15,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:52:15,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179425 s, 228 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:52:16,021::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:52:16,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147266 s, 278 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:52:25,899::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:52:25,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172851 s, 237 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:52:26,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:52:26,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00270441 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:52:35,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:52:36,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146478 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:52:36,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:52:36,238::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146233 s, 2.8 MB/s\n'; <rc> = 0
>Thread-15008::WARNING::2013-03-18 04:52:43,921::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:52:43,935::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:52:46,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:52:46,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866828 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:52:46,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:52:46,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851758 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:52:53,938::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:52:53,940::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:52:54,415::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:52:54,430::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:52:54,432::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:52:56,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:52:56,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121866 s, 336 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:52:56,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:52:56,382::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779899 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:53:06,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:53:06,132::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875982 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:53:06,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:53:06,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162051 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:53:16,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:53:16,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136788 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:53:16,507::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:53:16,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085517 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:53:26,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:53:26,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160641 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:53:26,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:53:26,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886695 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:53:36,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:53:36,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780007 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:53:36,648::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:53:36,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120874 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:53:46,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:53:46,311::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157941 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:53:46,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:53:46,793::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855797 s, 4.8 MB/s\n'; <rc> = 0
>Thread-15009::WARNING::2013-03-18 04:53:54,472::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:53:54,476::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:53:56,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:53:56,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163252 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:53:56,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:53:56,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00115531 s, 3.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:54:04,478::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:54:04,479::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:54:04,883::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:54:04,896::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:54:04,897::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:54:06,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:54:06,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184755 s, 222 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:54:06,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:54:06,950::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909696 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:54:16,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:54:16,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771967 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:54:16,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:54:17,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916627 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:54:26,451::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:54:26,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131309 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:54:27,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:54:27,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156507 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:54:36,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:54:36,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133751 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:54:37,147::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:54:37,175::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856555 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:54:46,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:54:46,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000735962 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:54:47,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:54:47,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132928 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:54:56,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:54:56,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132997 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:54:57,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:54:57,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000629393 s, 6.5 MB/s\n'; <rc> = 0
>Thread-15010::WARNING::2013-03-18 04:55:04,925::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:55:04,931::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:55:06,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:55:06,667::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840666 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:55:07,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:55:07,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151337 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:55:14,934::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:55:14,935::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:55:15,357::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:55:15,364::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:55:15,365::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:55:16,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:55:16,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163721 s, 250 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:55:17,465::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:55:17,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0018906 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:55:26,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:55:26,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00193104 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:55:27,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:55:27,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788626 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:55:36,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:55:36,809::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844821 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:55:37,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:55:37,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788715 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:55:46,814::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:55:46,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746192 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:55:47,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:55:47,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165054 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:55:56,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:55:56,899::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141765 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:55:57,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:55:57,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147343 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:56:06,904::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:56:06,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869432 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:56:07,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:56:07,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847948 s, 4.8 MB/s\n'; <rc> = 0
>Thread-15011::WARNING::2013-03-18 04:56:15,394::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:56:15,399::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:56:16,944::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:56:16,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153986 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:56:17,950::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:56:17,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795419 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:56:25,401::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 04:56:25,402::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:56:25,403::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:56:25,808::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:56:25,817::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:56:25,819::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:56:27,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:56:27,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170132 s, 241 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:56:28,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:56:28,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00429327 s, 954 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:56:37,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:56:37,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853077 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:56:38,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:56:38,130::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873049 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:56:47,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:56:47,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869008 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:56:48,174::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 04:56:48,175::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 04:56:48,176::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:56:48,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00278475 s, 735 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:56:48,213::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00278475 s, 735 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 04:56:48,213::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 04:56:48,213::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 04:56:48,215::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 04:56:48,216::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:56:48,217::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:56:48,217::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:56:48,218::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:56:48,218::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 04:56:48,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:56:48,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:56:48,658::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 04:56:48,659::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 04:56:48,662::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 04:56:48,663::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 04:56:48,664::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 04:56:48,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:56:49,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:56:49,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:56:49,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00854309 s, 479 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:56:57,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:56:57,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143161 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:56:59,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:56:59,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013422 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:57:07,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:57:07,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013796 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:57:09,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:57:09,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142883 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:57:17,216::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 04:57:17,217::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 04:57:17,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:57:17,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00135634 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:57:17,259::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00135634 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 04:57:17,260::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 04:57:17,261::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 04:57:17,262::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 04:57:17,263::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:57:17,264::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:57:17,264::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:57:17,264::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:57:17,265::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 04:57:17,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:57:17,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:57:17,700::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:57:17,701::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 04:57:17,703::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 04:57:17,704::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 04:57:17,705::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 04:57:17,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:57:18,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:57:18,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:57:18,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148252 s, 276 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:57:19,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:57:19,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161238 s, 254 kB/s\n'; <rc> = 0
>Thread-15012::WARNING::2013-03-18 04:57:25,852::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:57:25,858::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:57:28,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:57:28,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101492 s, 404 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:57:29,373::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:57:29,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00216889 s, 1.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:57:35,868::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:57:35,870::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:57:36,288::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:57:36,295::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:57:36,296::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:57:38,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:57:38,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111862 s, 366 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:57:39,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:57:39,511::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142682 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:57:48,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:57:48,299::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017614 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:57:49,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:57:49,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139908 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:57:58,305::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:57:58,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000975677 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:57:59,625::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:57:59,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791407 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:58:08,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:58:08,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924878 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:58:09,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:58:09,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150773 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:58:18,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:58:18,408::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077601 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:58:19,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:58:19,809::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131896 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:58:28,413::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:58:28,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810787 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:58:29,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:58:29,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784441 s, 5.2 MB/s\n'; <rc> = 0
>Thread-15013::WARNING::2013-03-18 04:58:36,325::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:58:36,329::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:58:38,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:58:38,484::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876799 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:58:39,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:58:39,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104973 s, 3.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:58:46,331::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:58:46,332::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:58:46,727::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:58:46,734::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:58:46,735::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:58:48,490::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:58:48,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167938 s, 244 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:58:49,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:58:50,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00225126 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:58:58,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:58:58,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802889 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:59:00,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:59:00,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142059 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:59:08,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:59:08,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141175 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:59:10,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:59:10,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142128 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:59:18,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:59:18,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820333 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:59:20,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:59:20,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100517 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:59:28,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:59:28,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873845 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:59:30,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:59:30,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150873 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 04:59:38,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:59:38,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134211 s, 305 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:59:40,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:59:40,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163784 s, 2.5 MB/s\n'; <rc> = 0
>Thread-15014::WARNING::2013-03-18 04:59:46,764::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 04:59:46,769::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 04:59:48,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:59:48,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111056 s, 369 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 04:59:50,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 04:59:50,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00986288 s, 415 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 04:59:56,773::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 04:59:56,774::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 04:59:57,191::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 04:59:57,198::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 04:59:57,199::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 04:59:58,922::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 04:59:58,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160505 s, 255 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:00:00,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:00:00,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792458 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:00:08,991::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:00:09,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00263746 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:00:10,606::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:00:10,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00097874 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:00:19,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:00:19,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00191032 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:00:20,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:00:20,700::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892474 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:00:29,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:00:29,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155212 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:00:30,749::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:00:30,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832281 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:00:39,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:00:39,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825206 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:00:40,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:00:40,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0030209 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:00:49,209::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:00:49,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141973 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:00:50,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:00:50,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000931096 s, 4.4 MB/s\n'; <rc> = 0
>Thread-15015::WARNING::2013-03-18 05:00:57,226::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:00:57,231::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:00:59,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:00:59,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815205 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:01:00,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:01:01,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156293 s, 2.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:01:07,233::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:01:07,234::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:01:07,745::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:01:07,752::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:01:07,753::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:01:09,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:01:09,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109923 s, 373 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:01:11,048::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:01:11,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867354 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:01:19,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:01:19,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159706 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:01:21,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:01:21,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892238 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:01:29,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:01:29,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000952101 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:01:31,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:01:31,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172406 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:01:39,423::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:01:39,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140419 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:01:41,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:01:41,297::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101745 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:01:49,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:01:49,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148346 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:01:51,343::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 05:01:51,345::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 05:01:51,345::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:01:51,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0432134 s, 47.4 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:01:51,423::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0432134 s, 47.4 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 05:01:51,423::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 05:01:51,423::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 05:01:51,425::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 05:01:51,426::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:01:51,427::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:01:51,427::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:01:51,427::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:01:51,428::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:01:51,429::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:01:51,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:01:51,880::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:01:51,881::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 05:01:51,884::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 05:01:51,886::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 05:01:51,886::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 05:01:51,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:01:52,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:01:52,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:01:52,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0318008 s, 129 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:01:59,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:01:59,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014573 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:02:02,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:02:02,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017754 s, 2.3 MB/s\n'; <rc> = 0
>Thread-15016::WARNING::2013-03-18 05:02:07,780::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:02:07,788::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:02:09,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:02:09,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805464 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:02:12,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:02:12,458::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803163 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:02:17,790::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 05:02:17,791::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:02:17,792::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:02:18,204::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:02:18,211::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:02:18,212::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:02:19,592::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 05:02:19,597::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 05:02:19,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:02:19,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0144198 s, 142 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:02:19,642::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0144198 s, 142 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 05:02:19,643::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 05:02:19,643::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 05:02:19,645::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 05:02:19,646::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:02:19,647::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:02:19,647::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:02:19,647::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:02:19,648::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:02:19,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:02:20,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:02:20,014::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:02:20,015::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 05:02:20,017::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 05:02:20,018::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 05:02:20,019::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 05:02:20,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:02:20,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:02:20,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:02:20,432::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0259312 s, 158 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:02:22,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:02:22,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132531 s, 309 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:02:30,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:02:30,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000630797 s, 6.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:02:32,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:02:32,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121836 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:02:40,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:02:40,513::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916803 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:02:42,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:02:42,683::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087645 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:02:50,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:02:50,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853909 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:02:52,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:02:52,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773666 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:03:00,556::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:03:00,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100222 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:03:02,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:03:02,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161702 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:03:10,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:03:10,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138167 s, 296 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:03:12,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:03:12,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123638 s, 3.3 MB/s\n'; <rc> = 0
>Thread-15017::WARNING::2013-03-18 05:03:18,241::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:03:18,249::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:03:20,647::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:03:20,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828523 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:03:22,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:03:22,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0023795 s, 1.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:03:28,261::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:03:28,262::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:03:28,664::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:03:28,671::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:03:28,673::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:03:30,681::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:03:30,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0196478 s, 208 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:03:33,021::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:03:33,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154467 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:03:40,740::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:03:40,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137828 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:03:43,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:03:43,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088038 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:03:50,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:03:50,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145872 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:03:53,171::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:03:53,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852681 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:04:00,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:04:00,869::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156906 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:04:03,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:04:03,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154737 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:04:10,874::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:04:10,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126388 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:04:13,324::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:04:13,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844131 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:04:20,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:04:20,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873977 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:04:23,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:04:23,439::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000889895 s, 4.6 MB/s\n'; <rc> = 0
>Thread-15018::WARNING::2013-03-18 05:04:28,696::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:04:28,700::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:04:30,959::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:04:30,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764717 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:04:33,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:04:33,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153596 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:04:38,701::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:04:38,702::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:04:39,133::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:04:39,140::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:04:39,142::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:04:40,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:04:41,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138095 s, 297 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:04:43,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:04:43,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134425 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:04:51,048::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:04:51,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780783 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:04:53,648::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:04:53,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0020059 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:05:01,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:05:01,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164526 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:05:03,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:05:03,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107655 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:05:11,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:05:11,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013679 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:05:13,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:05:13,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146756 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:05:21,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:05:21,231::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154414 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:05:23,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:05:23,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142043 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:05:31,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:05:31,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894934 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:05:33,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:05:34,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015176 s, 2.7 MB/s\n'; <rc> = 0
>Thread-15019::WARNING::2013-03-18 05:05:39,170::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:05:39,181::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:05:41,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:05:41,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932223 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:05:44,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:05:44,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00975181 s, 420 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:05:49,191::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:05:49,192::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:05:49,717::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:05:49,724::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:05:49,725::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:05:51,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:05:51,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134944 s, 304 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:05:54,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:05:54,163::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883437 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:06:01,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:06:01,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907109 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:06:04,209::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:06:04,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149285 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:06:11,401::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:06:11,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107593 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:06:14,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:06:14,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135759 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:06:21,435::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:06:21,482::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156138 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:06:24,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:06:24,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011017 s, 372 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:06:31,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:06:31,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143444 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:06:34,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:06:34,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818592 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:06:41,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:06:41,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904304 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:06:44,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:06:44,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108547 s, 377 kB/s\n'; <rc> = 0
>Thread-15020::WARNING::2013-03-18 05:06:49,756::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:06:49,760::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:06:51,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:06:51,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875374 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:06:54,648::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 05:06:54,650::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 05:06:54,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:06:54,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0011 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:06:54,692::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0011 s, 1.9 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 05:06:54,693::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 05:06:54,693::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 05:06:54,695::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 05:06:54,696::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:06:54,696::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:06:54,697::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:06:54,697::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:06:54,697::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:06:54,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:06:55,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:06:55,129::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:06:55,130::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 05:06:55,133::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 05:06:55,134::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 05:06:55,135::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 05:06:55,136::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:06:55,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:06:55,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:06:55,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143176 s, 286 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:06:59,762::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:06:59,763::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:07:00,190::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:07:00,197::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:07:00,199::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:07:01,612::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:07:01,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165594 s, 247 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:07:05,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:07:05,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894316 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:07:11,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:07:11,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00376703 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:07:15,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:07:15,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135617 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:07:21,708::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 05:07:21,709::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 05:07:21,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:07:21,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00074403 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:07:21,737::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00074403 s, 2.8 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 05:07:21,739::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 05:07:21,741::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 05:07:21,742::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 05:07:21,743::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:07:21,744::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:07:21,744::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:07:21,745::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:07:21,745::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:07:21,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:07:22,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:07:22,207::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:07:22,208::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 05:07:22,211::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 05:07:22,212::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 05:07:22,212::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 05:07:22,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:07:22,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:07:22,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:07:22,691::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014683 s, 279 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:07:25,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:07:25,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116858 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:07:32,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:07:32,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00099618 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:07:35,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:07:35,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792982 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:07:42,732::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:07:42,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898691 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:07:45,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:07:45,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138838 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:07:52,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:07:52,801::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863788 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:07:55,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:07:56,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147391 s, 2.8 MB/s\n'; <rc> = 0
>Thread-15021::WARNING::2013-03-18 05:08:00,233::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:08:00,236::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:08:02,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:08:02,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844557 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:08:06,061::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:08:06,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151572 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:08:10,237::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 05:08:10,238::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:08:10,239::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:08:10,675::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:08:10,681::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:08:10,682::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:08:12,840::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:08:12,882::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146625 s, 279 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:08:16,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:08:16,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893393 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:08:22,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:08:22,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014837 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:08:26,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:08:26,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132848 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:08:32,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:08:32,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015441 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:08:36,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:08:36,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772861 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:08:42,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:08:43,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849216 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:08:46,360::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:08:46,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859533 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:08:53,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:08:53,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160647 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:08:56,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:08:56,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767828 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:09:03,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:09:03,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744684 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:09:06,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:09:06,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859676 s, 4.8 MB/s\n'; <rc> = 0
>Thread-15022::WARNING::2013-03-18 05:09:10,709::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:09:10,714::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:09:13,108::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:09:13,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899074 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:09:16,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:09:16,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777881 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:09:20,719::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:09:20,720::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:09:21,149::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:09:21,156::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:09:21,157::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:09:23,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:09:23,188::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016076 s, 255 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:09:26,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:09:26,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783536 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:09:33,194::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:09:33,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767884 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:09:36,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:09:36,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154716 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:09:43,236::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:09:43,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014672 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:09:46,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:09:46,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014358 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:09:53,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:09:53,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088734 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:09:56,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:09:56,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000952884 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:10:03,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:10:03,364::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125702 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:10:06,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:10:07,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0331209 s, 124 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:10:13,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:10:13,401::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777071 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:10:17,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:10:17,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000955697 s, 4.3 MB/s\n'; <rc> = 0
>Thread-15023::WARNING::2013-03-18 05:10:21,184::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:10:21,189::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:10:23,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:10:23,434::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000940615 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:10:27,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:10:27,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000908791 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:10:31,191::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:10:31,192::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:10:31,596::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:10:31,603::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:10:31,604::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:10:33,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:10:33,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0196364 s, 209 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:10:37,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:10:37,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869326 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:10:43,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:10:43,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832562 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:10:47,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:10:47,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863392 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:10:53,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:10:53,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876001 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:10:57,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:10:57,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164444 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:11:03,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:11:03,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000991005 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:11:07,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:11:07,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161821 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:11:13,606::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:11:13,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169714 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:11:17,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:11:17,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148035 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:11:23,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:11:23,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771769 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:11:27,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:11:27,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755366 s, 5.4 MB/s\n'; <rc> = 0
>Thread-15024::WARNING::2013-03-18 05:11:31,632::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:11:31,635::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:11:33,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:11:33,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000929312 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:11:37,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:11:37,749::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00178002 s, 2.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:11:41,637::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:11:41,638::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:11:42,062::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:11:42,072::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:11:42,073::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:11:43,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:11:43,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173635 s, 236 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:11:47,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:11:47,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792698 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:11:53,760::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:11:53,786::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830387 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:11:57,855::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 05:11:57,857::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 05:11:57,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:11:57,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000889606 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:11:57,887::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000889606 s, 2.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 05:11:57,888::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 05:11:57,888::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 05:11:57,890::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 05:11:57,891::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:11:57,892::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:11:57,893::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:11:57,893::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:11:57,894::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:11:57,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:11:58,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:11:58,298::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:11:58,299::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 05:11:58,302::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 05:11:58,303::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 05:11:58,303::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 05:11:58,305::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:11:58,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:11:58,712::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:11:58,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130926 s, 313 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:12:03,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:12:03,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791921 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:12:08,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:12:08,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000938129 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:12:13,826::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:12:13,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147953 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:12:18,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:12:18,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00616892 s, 664 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:12:23,869::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 05:12:23,871::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 05:12:23,871::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:12:23,912::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00139264 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:12:23,914::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00139264 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 05:12:23,915::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 05:12:23,915::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 05:12:23,916::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 05:12:23,918::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:12:23,918::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:12:23,919::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:12:23,919::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:12:23,919::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:12:23,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:12:24,347::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:12:24,351::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:12:24,352::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 05:12:24,356::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 05:12:24,357::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 05:12:24,358::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 05:12:24,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:12:24,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:12:24,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:12:24,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108385 s, 378 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:12:28,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:12:28,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119632 s, 342 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:12:34,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:12:34,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103347 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:12:39,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:12:39,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140939 s, 2.9 MB/s\n'; <rc> = 0
>Thread-15025::WARNING::2013-03-18 05:12:42,096::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:12:42,101::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:12:44,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:12:44,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000989424 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:12:49,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:12:49,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139018 s, 2.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:12:52,102::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:12:52,103::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:12:52,544::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:12:52,551::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:12:52,552::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:12:54,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:12:54,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178482 s, 229 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:12:59,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:12:59,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912605 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:13:04,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:13:04,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100256 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:13:09,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:13:09,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172819 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:13:14,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:13:15,034::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00198487 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:13:19,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:13:19,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137318 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:13:25,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:13:25,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000933071 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:13:29,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:13:29,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137389 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:13:35,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:13:35,103::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835739 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:13:39,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:13:39,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149484 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:13:45,108::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:13:45,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837654 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:13:49,651::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:13:49,694::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000998805 s, 4.1 MB/s\n'; <rc> = 0
>Thread-15026::WARNING::2013-03-18 05:13:52,578::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:13:52,582::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:13:55,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:13:55,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076644 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:13:59,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:13:59,778::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148878 s, 2.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:14:02,583::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 05:14:02,584::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:14:02,585::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:14:03,028::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:14:03,035::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:14:03,036::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:14:05,184::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:14:05,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00883735 s, 463 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:14:09,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:14:09,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159979 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:14:15,230::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:14:15,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164635 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:14:19,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:14:19,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144937 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:14:25,280::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:14:25,311::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00638246 s, 642 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:14:29,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:14:30,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823903 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:14:35,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:14:35,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879568 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:14:40,045::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:14:40,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144799 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:14:45,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:14:45,402::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133773 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:14:50,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:14:50,163::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158495 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:14:55,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:14:55,439::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777086 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:15:00,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:15:00,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00436987 s, 937 kB/s\n'; <rc> = 0
>Thread-15027::WARNING::2013-03-18 05:15:03,064::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:15:03,068::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:15:05,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:15:05,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822859 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:15:10,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:15:10,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000934543 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:15:13,070::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:15:13,071::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:15:13,502::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:15:13,509::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:15:13,510::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:15:15,481::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:15:15,521::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126209 s, 325 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:15:20,366::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:15:20,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771851 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:15:25,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:15:25,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875881 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:15:30,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:15:30,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843294 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:15:35,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:15:35,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781144 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:15:40,508::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:15:40,548::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154804 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:15:45,606::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:15:45,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156218 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:15:50,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:15:50,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150682 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:15:55,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:15:55,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142221 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:16:00,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:16:00,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0162015 s, 253 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:16:05,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:16:05,738::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740109 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:16:10,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:16:10,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776644 s, 5.3 MB/s\n'; <rc> = 0
>Thread-15028::WARNING::2013-03-18 05:16:13,537::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:16:13,541::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:16:15,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:16:15,771::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792502 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:16:20,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:16:20,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017211 s, 2.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:16:23,543::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:16:23,544::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:16:23,977::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:16:23,984::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:16:23,985::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:16:25,778::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:16:25,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169989 s, 241 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:16:30,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:16:30,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776456 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:16:35,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:16:35,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128476 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:16:41,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:16:41,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157638 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:16:45,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:16:45,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925086 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:16:51,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:16:51,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755822 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:16:55,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:16:55,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819056 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:17:01,181::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 05:17:01,182::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 05:17:01,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:17:01,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000813988 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:17:01,216::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000813988 s, 2.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 05:17:01,217::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 05:17:01,217::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 05:17:01,219::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 05:17:01,221::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:17:01,222::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:17:01,222::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:17:01,223::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:17:01,223::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:17:01,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:17:01,658::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:17:01,662::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:17:01,663::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 05:17:01,666::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 05:17:01,667::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 05:17:01,668::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 05:17:01,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:17:02,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:17:02,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:17:02,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0090078 s, 455 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:17:05,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:17:05,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125493 s, 326 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:17:12,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:17:12,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134212 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:17:15,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:17:16,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813479 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:17:22,188::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:17:22,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126906 s, 3.2 MB/s\n'; <rc> = 0
>Thread-15029::WARNING::2013-03-18 05:17:24,021::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:17:24,028::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:17:26,031::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 05:17:26,032::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 05:17:26,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:17:26,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000846196 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:17:26,060::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000846196 s, 2.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 05:17:26,060::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 05:17:26,060::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 05:17:26,062::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 05:17:26,063::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:17:26,064::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:17:26,064::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:17:26,064::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:17:26,065::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:17:26,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:17:26,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:17:26,507::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:17:26,508::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 05:17:26,510::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 05:17:26,511::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 05:17:26,512::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 05:17:26,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:17:26,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:17:26,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:17:26,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.012594 s, 325 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:17:32,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:17:32,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00859655 s, 476 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:17:34,067::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:17:34,069::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:17:34,474::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:17:34,481::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:17:34,482::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:17:36,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:17:36,963::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102898 s, 398 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:17:42,382::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:17:42,408::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086835 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:17:46,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:17:47,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171146 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:17:52,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:17:52,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140437 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:17:57,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:17:57,046::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00092448 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:18:02,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:18:02,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804454 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:18:07,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:18:07,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083876 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:18:12,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:18:12,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146999 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:18:17,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:18:17,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149131 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:18:22,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:18:22,709::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000926546 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:18:27,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:18:27,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907147 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:18:32,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:18:32,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074528 s, 5.5 MB/s\n'; <rc> = 0
>Thread-15030::WARNING::2013-03-18 05:18:34,508::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:18:34,512::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:18:37,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:18:37,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891046 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:18:42,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:18:42,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000695479 s, 5.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:18:44,521::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:18:44,522::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:18:44,971::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:18:44,978::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:18:44,979::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:18:47,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:18:47,260::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173688 s, 236 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:18:52,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:18:52,971::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763032 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:18:57,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:18:57,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842214 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:19:03,014::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:19:03,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791945 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:19:07,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:19:07,331::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797368 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:19:13,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:19:13,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769744 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:19:17,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:19:17,366::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835157 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:19:23,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:19:23,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133726 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:19:27,371::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:19:27,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146536 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:19:33,227::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:19:33,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135142 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:19:37,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:19:37,471::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140828 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:19:43,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:19:43,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00477737 s, 857 kB/s\n'; <rc> = 0
>Thread-15031::WARNING::2013-03-18 05:19:45,005::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:19:45,009::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:19:47,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:19:47,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860398 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:19:53,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:19:53,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000747073 s, 5.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:19:55,011::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 05:19:55,012::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:19:55,013::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:19:55,435::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:19:55,441::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:19:55,443::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:19:57,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:19:57,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123925 s, 331 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:20:03,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:20:03,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167394 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:20:07,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:20:07,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086258 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:20:13,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:20:13,612::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145868 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:20:17,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:20:17,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874107 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:20:23,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:20:23,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139977 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:20:27,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:20:27,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013566 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:20:33,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:20:33,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00901868 s, 454 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:20:37,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:20:37,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133145 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:20:43,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:20:43,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810701 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:20:47,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:20:47,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120149 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:20:53,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:20:53,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105549 s, 3.9 MB/s\n'; <rc> = 0
>Thread-15032::WARNING::2013-03-18 05:20:55,469::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:20:55,473::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:20:57,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:20:57,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152718 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:21:04,203::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:21:04,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0267316 s, 153 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:21:05,481::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:21:05,482::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:21:05,924::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:21:05,931::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:21:05,932::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:21:07,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:21:07,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129597 s, 316 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:21:14,312::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:21:14,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838572 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:21:17,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:21:17,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156925 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:21:24,383::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:21:24,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822836 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:21:27,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:21:27,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829423 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:21:34,452::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:21:34,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147751 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:21:37,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:21:37,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109779 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:21:44,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:21:44,587::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140344 s, 292 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:21:47,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:21:48,011::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000926523 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:21:54,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:21:54,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140383 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:21:58,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:21:58,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136069 s, 301 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:22:04,716::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 05:22:04,724::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 05:22:04,724::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:22:04,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000718079 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:22:04,786::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000718079 s, 2.9 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 05:22:04,786::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 05:22:04,787::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 05:22:04,789::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 05:22:04,790::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:22:04,791::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:22:04,792::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:22:04,792::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:22:04,808::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:22:04,848::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:22:05,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:22:05,619::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:22:05,624::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 05:22:05,636::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 05:22:05,640::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 05:22:05,642::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 05:22:05,648::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-15033::WARNING::2013-03-18 05:22:05,972::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:22:05,979::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 05:22:06,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:22:06,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:22:06,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0157317 s, 260 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:22:08,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:22:08,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156121 s, 262 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:22:15,981::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:22:15,982::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:22:16,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:22:16,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122761 s, 334 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:22:16,433::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:22:16,438::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:22:16,439::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:22:18,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:22:18,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110882 s, 369 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:22:26,254::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:22:26,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131448 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:22:28,165::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 05:22:28,166::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 05:22:28,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:22:28,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000803032 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:22:28,195::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000803032 s, 2.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 05:22:28,196::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 05:22:28,196::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 05:22:28,198::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 05:22:28,199::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:22:28,199::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:22:28,200::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:22:28,200::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:22:28,200::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:22:28,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:22:28,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:22:28,634::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:22:28,635::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 05:22:28,637::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 05:22:28,639::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 05:22:28,639::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 05:22:28,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:22:29,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:22:29,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:22:29,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.017059 s, 240 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:22:36,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:22:36,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130597 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:22:39,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:22:39,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0276245 s, 148 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:22:46,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:22:46,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791554 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:22:49,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:22:49,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150829 s, 272 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:22:56,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:22:56,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014483 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:22:59,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:22:59,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147851 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:23:06,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:23:06,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014435 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:23:09,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:23:09,334::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175394 s, 2.3 MB/s\n'; <rc> = 0
>Thread-15034::WARNING::2013-03-18 05:23:16,466::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:23:16,470::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 05:23:16,643::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:23:16,667::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793774 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:23:19,345::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:23:19,374::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923995 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:23:26,472::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:23:26,473::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:23:26,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:23:26,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112575 s, 364 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:23:26,950::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:23:26,955::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:23:26,956::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:23:29,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:23:29,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013152 s, 311 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:23:36,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:23:36,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875999 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:23:39,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:23:39,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106802 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:23:46,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:23:46,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0382416 s, 107 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:23:49,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:23:49,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127258 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:23:57,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:23:57,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883854 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:23:59,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:23:59,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824309 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:24:07,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:24:07,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131456 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:24:09,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:24:09,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110006 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:24:17,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:24:17,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150558 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:24:19,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:24:19,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091589 s, 4.5 MB/s\n'; <rc> = 0
>Thread-15035::WARNING::2013-03-18 05:24:26,991::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:24:26,997::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 05:24:27,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:24:27,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151248 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:24:29,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:24:29,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832093 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:24:37,000::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:24:37,001::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:24:37,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:24:37,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00917629 s, 446 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:24:37,434::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:24:37,466::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:24:37,471::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:24:39,782::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:24:39,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00908201 s, 451 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:24:47,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:24:47,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00213641 s, 1.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:24:49,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:24:49,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136735 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:24:57,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:24:57,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000703783 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:24:59,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:24:59,936::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00525049 s, 780 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:25:07,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:25:07,686::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153656 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:25:09,941::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:25:09,979::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818239 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:25:17,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:25:17,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122923 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:25:19,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:25:20,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00208779 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:25:27,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:25:27,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137828 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:25:30,034::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:25:30,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000965046 s, 4.2 MB/s\n'; <rc> = 0
>Thread-15036::WARNING::2013-03-18 05:25:37,521::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:25:37,523::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 05:25:37,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:25:37,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00193602 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:25:40,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:25:40,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897785 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:25:47,525::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 05:25:47,526::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:25:47,527::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:25:47,952::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:25:47,959::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:25:47,960::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:25:47,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:25:48,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127059 s, 322 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:25:50,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:25:50,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206249 s, 199 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:25:58,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:25:58,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810671 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:26:00,194::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:26:00,222::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083401 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:26:08,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:26:08,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139198 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:26:10,227::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:26:10,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147133 s, 278 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:26:18,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:26:18,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808987 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:26:20,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:26:20,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146771 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:26:28,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:26:28,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015416 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:26:30,350::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:26:30,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110232 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:26:38,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:26:38,448::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838989 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:26:40,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:26:40,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128954 s, 3.2 MB/s\n'; <rc> = 0
>Thread-15037::WARNING::2013-03-18 05:26:48,029::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:26:48,032::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 05:26:48,533::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:26:48,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756916 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:26:50,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:26:50,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870288 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:26:58,034::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:26:58,036::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:26:58,460::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:26:58,467::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:26:58,468::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:26:58,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:26:58,658::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113851 s, 360 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:27:00,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:27:00,562::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128108 s, 320 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:27:08,696::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 05:27:08,698::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 05:27:08,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:27:08,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000807284 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:27:08,729::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000807284 s, 2.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 05:27:08,729::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 05:27:08,729::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 05:27:08,731::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 05:27:08,737::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:27:08,738::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:27:08,739::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:27:08,739::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:27:08,739::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:27:08,740::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:27:09,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:27:09,149::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:27:09,150::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 05:27:09,152::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 05:27:09,154::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 05:27:09,154::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 05:27:09,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:27:09,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:27:09,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:27:09,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00599133 s, 684 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:27:10,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:27:10,610::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148193 s, 276 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:27:19,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:27:19,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110341 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:27:20,617::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:27:20,657::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157471 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:27:29,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:27:29,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013372 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:27:30,661::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 05:27:30,662::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 05:27:30,663::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:27:30,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000716871 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:27:30,691::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000716871 s, 2.9 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 05:27:30,691::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 05:27:30,692::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 05:27:30,693::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 05:27:30,694::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:27:30,695::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:27:30,695::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:27:30,696::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:27:30,696::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:27:30,697::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:27:31,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:27:31,136::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:27:31,137::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 05:27:31,139::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 05:27:31,140::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 05:27:31,141::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 05:27:31,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:27:31,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:27:31,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:27:31,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153368 s, 267 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:27:39,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:27:39,802::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013498 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:27:41,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:27:41,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146094 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:27:49,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:27:49,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141584 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:27:51,625::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:27:51,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164217 s, 249 kB/s\n'; <rc> = 0
>Thread-15038::WARNING::2013-03-18 05:27:58,494::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:27:58,501::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 05:27:59,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:27:59,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835153 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:28:01,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:28:01,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156881 s, 2.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:28:08,505::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:28:08,506::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:28:08,928::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:28:08,934::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:28:08,935::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:28:09,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:28:10,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00873338 s, 469 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:28:11,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:28:11,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125516 s, 326 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:28:20,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:28:20,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761004 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:28:21,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:28:21,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877882 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:28:30,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:28:30,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128802 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:28:31,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:28:31,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156349 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:28:40,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:28:40,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079767 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:28:41,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:28:41,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833988 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:28:50,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:28:50,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759855 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:28:51,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:28:51,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161522 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:29:00,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:29:00,392::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840084 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:29:01,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:29:02,003::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860978 s, 4.8 MB/s\n'; <rc> = 0
>Thread-15039::WARNING::2013-03-18 05:29:08,971::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:29:08,975::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 05:29:10,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:29:10,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117033 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:29:12,008::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:29:12,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084316 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:29:18,977::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:29:18,978::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:29:19,398::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:29:19,405::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:29:19,406::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:29:20,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:29:20,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142556 s, 287 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:29:22,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:29:22,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192201 s, 213 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:29:30,605::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:29:30,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133806 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:29:32,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:29:32,129::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811673 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:29:40,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:29:40,719::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768024 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:29:42,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:29:42,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016949 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:29:50,760::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:29:50,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150174 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:29:52,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:29:52,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792372 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:30:00,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:30:00,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138137 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:30:02,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:30:02,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839476 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:30:10,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:30:11,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861804 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:30:12,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:30:12,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206481 s, 198 kB/s\n'; <rc> = 0
>Thread-15040::WARNING::2013-03-18 05:30:19,436::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:30:19,439::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 05:30:21,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:30:21,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853675 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:30:22,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:30:22,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142207 s, 2.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:30:29,442::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:30:29,443::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:30:29,905::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:30:29,912::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:30:29,913::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:30:31,118::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:30:31,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118598 s, 345 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:30:32,347::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:30:32,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00731336 s, 560 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:30:41,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:30:41,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772565 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:30:42,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:30:42,424::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104451 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:30:51,271::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:30:51,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771041 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:30:52,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:30:52,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101965 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:31:01,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:31:01,368::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000766067 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:31:02,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:31:02,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848105 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:31:11,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:31:11,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142644 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:31:12,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:31:12,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122864 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:31:21,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:31:21,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147954 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:31:22,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:31:22,588::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014785 s, 2.8 MB/s\n'; <rc> = 0
>Thread-15041::WARNING::2013-03-18 05:31:29,952::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:31:29,959::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 05:31:31,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:31:31,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792011 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:31:32,593::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:31:32,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799402 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:31:39,962::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 05:31:39,963::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:31:39,964::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:31:40,353::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:31:40,359::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:31:40,361::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:31:41,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:31:41,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00895898 s, 457 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:31:42,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:31:42,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165643 s, 247 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:31:51,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:31:51,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871388 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:31:52,685::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:31:52,733::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166942 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:32:01,835::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:32:01,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00312463 s, 1.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:32:02,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:32:02,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015304 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:32:11,921::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 05:32:11,922::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 05:32:11,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:32:11,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000819745 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:32:11,956::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000819745 s, 2.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 05:32:11,957::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 05:32:11,957::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 05:32:11,959::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 05:32:11,960::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:32:11,960::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:32:11,961::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:32:11,961::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:32:11,962::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:32:11,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:32:12,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:32:12,413::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:32:12,414::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 05:32:12,416::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 05:32:12,418::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 05:32:12,418::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 05:32:12,419::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:32:12,777::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:32:12,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:32:12,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:32:12,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185448 s, 221 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:32:12,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180882 s, 226 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:32:22,861::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:32:22,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016757 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:32:22,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:32:22,950::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138703 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:32:32,898::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 05:32:32,899::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 05:32:32,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:32:32,928::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000783311 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:32:32,934::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000783311 s, 2.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 05:32:32,934::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 05:32:32,934::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 05:32:32,936::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 05:32:32,937::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:32:32,938::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:32:32,938::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:32:32,938::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:32:32,939::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:32:32,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:32:32,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:32:33,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146338 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:32:33,383::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:32:33,387::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:32:33,388::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 05:32:33,390::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 05:32:33,391::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 05:32:33,392::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 05:32:33,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:32:33,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:32:33,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:32:33,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117824 s, 348 kB/s\n'; <rc> = 0
>Thread-15042::WARNING::2013-03-18 05:32:40,389::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:32:40,392::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 05:32:43,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:32:43,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769281 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:32:43,842::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:32:43,890::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797213 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:32:50,394::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:32:50,396::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:32:50,823::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:32:50,830::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:32:50,831::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:32:53,156::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:32:53,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156846 s, 261 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:32:53,895::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:32:53,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0171595 s, 239 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:33:03,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:33:03,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074787 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:33:03,967::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:33:04,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000971472 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:33:13,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:33:13,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087882 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:33:14,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:33:14,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111333 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:33:23,373::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:33:23,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871303 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:33:24,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:33:24,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00253832 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:33:33,439::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:33:33,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774423 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:33:34,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:33:34,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086774 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:33:43,510::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:33:43,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130518 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:33:44,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:33:44,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206741 s, 198 kB/s\n'; <rc> = 0
>Thread-15043::WARNING::2013-03-18 05:33:50,872::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:33:50,876::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 05:33:53,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:33:53,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000938174 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:33:54,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:33:54,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00193531 s, 2.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:34:00,883::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:34:00,884::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:34:01,397::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:34:01,404::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:34:01,406::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:34:03,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:34:03,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122849 s, 333 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:34:04,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:34:04,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199575 s, 205 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:34:13,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:34:13,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887594 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:34:14,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:34:14,424::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142142 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:34:23,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:34:23,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835112 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:34:24,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:34:24,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00286261 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:34:33,896::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:34:33,942::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158456 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:34:34,475::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:34:34,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117657 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:34:43,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:34:44,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192384 s, 213 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:34:44,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:34:44,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875435 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:34:54,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:34:54,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826229 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:34:54,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:34:54,645::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00096864 s, 4.2 MB/s\n'; <rc> = 0
>Thread-15044::WARNING::2013-03-18 05:35:01,433::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:35:01,437::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 05:35:04,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:35:04,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130014 s, 315 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:35:04,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:35:04,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786257 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:35:11,439::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:35:11,440::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:35:12,013::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:35:12,020::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:35:12,021::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:35:14,226::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:35:14,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0111786 s, 366 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:35:14,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:35:14,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142433 s, 288 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:35:24,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:35:24,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00201888 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:35:24,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:35:24,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784294 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:35:34,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:35:34,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105043 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:35:34,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:35:34,805::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016351 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:35:44,458::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:35:44,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153145 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:35:44,811::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:35:44,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140754 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:35:54,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:35:54,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814683 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:35:54,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:35:54,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142378 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:36:04,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:36:04,650::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148905 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:36:04,910::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:36:04,945::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148663 s, 2.8 MB/s\n'; <rc> = 0
>Thread-15045::WARNING::2013-03-18 05:36:12,048::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:36:12,054::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 05:36:14,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:36:14,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936615 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:36:14,951::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:36:14,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000969748 s, 4.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:36:22,063::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:36:22,064::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:36:22,562::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:36:22,568::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:36:22,570::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:36:24,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:36:24,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146185 s, 280 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:36:24,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:36:25,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181181 s, 226 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:36:34,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:36:34,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000925994 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:36:35,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:36:35,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834183 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:36:44,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:36:44,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117505 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:36:45,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:36:45,123::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013277 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:36:54,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:36:55,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758712 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:36:55,128::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:36:55,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139503 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:37:05,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:37:05,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826035 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:37:05,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:37:05,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144712 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:37:15,122::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 05:37:15,124::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 05:37:15,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:37:15,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00274692 s, 746 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:37:15,163::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00274692 s, 746 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 05:37:15,163::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 05:37:15,163::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 05:37:15,165::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 05:37:15,166::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:37:15,167::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:37:15,168::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:37:15,168::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:37:15,168::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:37:15,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:37:15,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:37:15,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157159 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:37:15,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:37:15,644::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:37:15,645::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 05:37:15,648::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 05:37:15,649::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 05:37:15,650::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 05:37:15,652::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:37:16,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:37:16,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:37:16,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126963 s, 323 kB/s\n'; <rc> = 0
>Thread-15046::WARNING::2013-03-18 05:37:22,597::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:37:22,602::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:37:25,289::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:37:25,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156241 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:37:26,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:37:26,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104107 s, 3.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:37:32,603::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 05:37:32,604::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:37:32,605::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:37:33,124::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:37:33,131::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:37:33,132::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:37:35,326::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 05:37:35,329::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 05:37:35,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:37:35,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0144159 s, 142 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:37:35,375::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0144159 s, 142 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 05:37:35,375::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 05:37:35,375::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 05:37:35,377::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 05:37:35,378::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:37:35,379::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:37:35,379::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:37:35,379::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:37:35,380::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:37:35,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:37:35,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:37:35,819::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:37:35,819::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 05:37:35,822::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 05:37:35,823::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 05:37:35,823::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 05:37:35,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:37:36,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:37:36,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:37:36,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:37:36,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178903 s, 229 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:37:36,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152227 s, 269 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:37:46,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:37:46,316::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:37:46,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145546 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:37:46,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169079 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:37:56,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:37:56,395::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156563 s, 262 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:37:56,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:37:56,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136791 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:38:06,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:38:06,456::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823648 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:38:06,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:38:06,526::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163068 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:38:16,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:38:16,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141304 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:38:16,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:38:16,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141437 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:38:26,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:38:26,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152369 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:38:26,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:38:26,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00290075 s, 1.4 MB/s\n'; <rc> = 0
>Thread-15047::WARNING::2013-03-18 05:38:33,172::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:38:33,178::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:38:36,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:38:36,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899056 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:38:36,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:38:36,756::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818939 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:38:43,180::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:38:43,181::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:38:43,687::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:38:43,697::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:38:43,699::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:38:46,589::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:38:46,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161263 s, 254 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:38:46,797::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:38:46,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114428 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:38:56,648::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:38:56,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849696 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:38:56,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:38:56,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741538 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:39:06,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:39:06,731::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00403768 s, 1.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:39:06,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:39:06,964::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824923 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:39:16,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:39:16,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149755 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:39:17,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:39:17,045::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146615 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:39:26,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:39:26,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150168 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:39:27,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:39:27,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786969 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:39:36,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:39:36,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160984 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:39:37,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:39:37,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149075 s, 2.7 MB/s\n'; <rc> = 0
>Thread-15048::WARNING::2013-03-18 05:39:43,725::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:39:43,730::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:39:46,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:39:46,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787131 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:39:47,244::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:39:47,270::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906844 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:39:53,733::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:39:53,734::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:39:54,209::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:39:54,215::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:39:54,216::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:39:56,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:39:56,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122298 s, 335 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:39:57,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:39:57,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000911466 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:40:06,965::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:40:06,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861114 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:40:07,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:40:07,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845064 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:40:17,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:40:17,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000871519 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:40:17,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:40:17,478::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825689 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:40:27,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:40:27,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00276215 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:40:27,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:40:27,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000987852 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:40:37,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:40:37,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086499 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:40:37,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:40:37,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144078 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:40:47,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:40:47,195::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101316 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:40:47,668::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:40:47,709::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139663 s, 2.9 MB/s\n'; <rc> = 0
>Thread-15049::WARNING::2013-03-18 05:40:54,243::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:40:54,246::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:40:57,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:40:57,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786379 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:40:57,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:40:57,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833429 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:41:04,254::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:41:04,255::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:41:04,734::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:41:04,741::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:41:04,742::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:41:07,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:41:07,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194854 s, 210 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:41:07,816::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:41:07,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771934 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:41:17,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:41:17,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902007 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:41:17,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:41:17,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136316 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:41:27,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:41:27,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00075975 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:41:27,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:41:28,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013858 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:41:37,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:41:37,390::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869818 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:41:38,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:41:38,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144867 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:41:47,396::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:41:47,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812721 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:41:48,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:41:48,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131781 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:41:57,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:41:57,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790383 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:41:58,235::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:41:58,279::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156246 s, 2.6 MB/s\n'; <rc> = 0
>Thread-15050::WARNING::2013-03-18 05:42:04,770::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:42:04,775::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:42:07,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:42:07,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140515 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:42:08,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:42:08,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000959674 s, 4.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:42:14,778::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:42:14,779::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:42:15,219::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:42:15,226::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:42:15,227::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:42:17,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:42:17,561::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118553 s, 345 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:42:18,397::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 05:42:18,399::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 05:42:18,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:42:18,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00165882 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:42:18,444::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00165882 s, 1.2 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 05:42:18,444::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 05:42:18,445::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 05:42:18,446::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 05:42:18,447::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:42:18,448::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:42:18,448::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:42:18,449::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:42:18,449::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:42:18,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:42:18,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:42:18,919::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:42:18,920::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 05:42:18,923::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 05:42:18,925::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 05:42:18,925::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 05:42:18,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:42:19,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:42:19,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:42:19,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793106 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:42:27,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:42:27,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796488 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:42:29,336::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:42:29,372::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129223 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:42:37,605::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 05:42:37,606::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 05:42:37,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:42:37,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00107935 s, 1.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:42:37,654::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00107935 s, 1.9 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 05:42:37,655::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 05:42:37,655::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 05:42:37,656::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 05:42:37,658::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:42:37,658::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:42:37,659::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:42:37,659::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:42:37,659::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:42:37,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:42:38,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:42:38,108::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:42:38,109::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 05:42:38,111::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 05:42:38,112::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 05:42:38,113::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 05:42:38,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:42:38,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:42:38,477::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:42:38,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0261747 s, 156 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:42:39,412::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:42:39,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00684323 s, 599 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:42:48,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:42:48,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738819 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:42:49,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:42:49,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000686572 s, 6.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:42:58,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:42:58,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0148831 s, 275 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:42:59,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:42:59,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014557 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:43:08,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:43:08,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00268367 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:43:09,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:43:09,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176859 s, 2.3 MB/s\n'; <rc> = 0
>Thread-15051::WARNING::2013-03-18 05:43:15,261::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:43:15,265::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:43:18,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:43:18,707::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810068 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:43:19,747::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:43:19,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746056 s, 5.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:43:25,267::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 05:43:25,268::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:43:25,269::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:43:25,695::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:43:25,702::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:43:25,704::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:43:28,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:43:28,760::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.012076 s, 339 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:43:29,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:43:29,843::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135645 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:43:38,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:43:38,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133571 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:43:39,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:43:39,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00222209 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:43:48,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:43:48,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853997 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:43:49,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:43:49,987::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00120642 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:43:58,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:43:58,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152089 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:44:00,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:44:00,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161511 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:44:08,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:44:08,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817646 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:44:10,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:44:10,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136747 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:44:18,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:44:18,950::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000952319 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:44:20,179::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:44:20,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111454 s, 3.7 MB/s\n'; <rc> = 0
>Thread-15052::WARNING::2013-03-18 05:44:25,731::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:44:25,737::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:44:28,956::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:44:28,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894183 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:44:30,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:44:30,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857364 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:44:35,740::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:44:35,741::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:44:36,236::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:44:36,242::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:44:36,244::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:44:38,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:44:39,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190751 s, 215 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:44:40,325::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:44:40,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0220619 s, 186 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:44:49,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:44:49,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796418 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:44:50,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:44:50,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000946277 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:44:59,079::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:44:59,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085605 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:45:00,492::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:45:00,521::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126208 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:45:09,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:45:09,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789176 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:45:10,560::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:45:10,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823022 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:45:19,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:45:19,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804685 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:45:20,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:45:20,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143589 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:45:29,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:45:29,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170974 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:45:30,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:45:30,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150468 s, 2.7 MB/s\n'; <rc> = 0
>Thread-15053::WARNING::2013-03-18 05:45:36,271::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:45:36,275::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:45:39,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:45:39,257::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878566 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:45:40,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:45:40,833::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000886886 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:45:46,277::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:45:46,278::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:45:46,714::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:45:46,721::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:45:46,723::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:45:49,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:45:49,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015968 s, 257 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:45:50,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:45:50,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00083915 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:45:59,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:45:59,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796077 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:46:00,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:46:00,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810689 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:46:09,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:46:09,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00232794 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:46:11,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:46:11,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00883696 s, 464 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:46:19,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:46:19,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140129 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:46:21,105::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:46:21,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131387 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:46:29,431::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:46:29,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774998 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:46:31,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:46:31,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146604 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:46:39,465::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:46:39,511::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162038 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:46:41,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:46:41,305::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074372 s, 5.5 MB/s\n'; <rc> = 0
>Thread-15054::WARNING::2013-03-18 05:46:46,752::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:46:46,757::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:46:49,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:46:49,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00099746 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:46:51,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:46:51,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899347 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:46:56,765::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:46:56,766::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:46:57,250::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:46:57,257::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:46:57,258::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:46:59,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:46:59,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152293 s, 269 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:47:01,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:47:01,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118763 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:47:09,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:47:09,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150354 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:47:11,487::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:47:11,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167716 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:47:19,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:47:19,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881318 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:47:21,569::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 05:47:21,571::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 05:47:21,571::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:47:21,611::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00126145 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:47:21,612::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00126145 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 05:47:21,613::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 05:47:21,613::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 05:47:21,615::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 05:47:21,616::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:47:21,616::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:47:21,617::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:47:21,617::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:47:21,618::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:47:21,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:47:22,029::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:47:22,033::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:47:22,034::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 05:47:22,037::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 05:47:22,038::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 05:47:22,038::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 05:47:22,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:47:22,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:47:22,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:47:22,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00776138 s, 528 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:47:29,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:47:29,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813004 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:47:32,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:47:32,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767505 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:47:39,730::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 05:47:39,732::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 05:47:39,732::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:47:39,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000765799 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:47:39,766::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000765799 s, 2.7 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 05:47:39,767::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 05:47:39,767::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 05:47:39,769::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 05:47:39,770::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:47:39,771::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:47:39,772::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:47:39,772::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:47:39,774::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:47:39,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:47:40,234::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:47:40,238::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:47:40,239::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 05:47:40,243::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 05:47:40,244::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 05:47:40,245::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 05:47:40,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:47:40,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:47:40,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:47:40,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00738404 s, 555 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:47:42,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:47:42,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115431 s, 355 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:47:50,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:47:50,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088635 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:47:52,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:47:52,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00306161 s, 1.3 MB/s\n'; <rc> = 0
>Thread-15055::WARNING::2013-03-18 05:47:57,293::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:47:57,296::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:48:00,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:48:00,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158199 s, 259 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:48:02,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:48:02,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793982 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:48:07,301::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:48:07,302::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:48:07,778::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:48:07,785::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:48:07,786::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:48:10,760::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:48:10,801::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158633 s, 258 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:48:12,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:48:12,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0221114 s, 185 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:48:20,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:48:20,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116579 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:48:22,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:48:22,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00172076 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:48:30,841::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:48:30,866::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867671 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:48:32,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:48:32,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882804 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:48:40,871::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:48:40,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775107 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:48:42,997::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:48:43,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100376 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:48:50,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:48:50,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738539 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:48:53,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:48:53,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156916 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:49:00,942::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:49:00,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082969 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:49:03,141::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:49:03,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138949 s, 2.9 MB/s\n'; <rc> = 0
>Thread-15056::WARNING::2013-03-18 05:49:07,812::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:49:07,815::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:49:10,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:49:11,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898737 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:49:13,221::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:49:13,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772948 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:49:17,819::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 05:49:17,820::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:49:17,821::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:49:18,253::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:49:18,259::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:49:18,261::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:49:21,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:49:21,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.01661 s, 247 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:49:23,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:49:23,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0084038 s, 487 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:49:31,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:49:31,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796698 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:49:33,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:49:33,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008691 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:49:41,104::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:49:41,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143794 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:49:43,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:49:43,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00207616 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:49:51,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:49:51,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136454 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:49:53,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:49:53,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013902 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:50:01,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:50:01,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761002 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:50:03,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:50:03,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138251 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:50:11,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:50:11,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141406 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:50:13,656::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:50:13,698::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173781 s, 2.4 MB/s\n'; <rc> = 0
>Thread-15057::WARNING::2013-03-18 05:50:18,289::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:50:18,294::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:50:21,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:50:21,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0437559 s, 93.6 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:50:23,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:50:23,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000741064 s, 5.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:50:28,296::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:50:28,297::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:50:28,748::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:50:28,754::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:50:28,756::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:50:31,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:50:31,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167048 s, 245 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:50:33,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:50:33,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163729 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:50:41,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:50:41,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079417 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:50:43,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:50:43,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819745 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:50:51,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:50:51,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844878 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:50:53,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:50:54,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159973 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:51:01,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:51:01,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142165 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:51:04,079::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:51:04,130::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014253 s, 287 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:51:11,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:51:11,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111725 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:51:14,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:51:14,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0017254 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:51:21,605::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:51:21,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127857 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:51:24,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:51:24,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106973 s, 383 kB/s\n'; <rc> = 0
>Thread-15058::WARNING::2013-03-18 05:51:28,783::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:51:28,791::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:51:31,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:51:31,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796953 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:51:34,352::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:51:34,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758121 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:51:38,801::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:51:38,802::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:51:39,276::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:51:39,282::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:51:39,284::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:51:41,682::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:51:41,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167812 s, 244 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:51:44,415::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:51:44,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00697148 s, 588 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:51:51,732::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:51:51,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873902 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:51:54,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:51:54,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920429 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:52:01,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:52:01,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015385 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:52:04,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:52:04,617::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100826 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:52:11,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:52:11,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000927795 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:52:14,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:52:14,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793116 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:52:21,868::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:52:21,910::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014441 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:52:24,727::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 05:52:24,728::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 05:52:24,729::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:52:24,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00140164 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:52:24,771::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00140164 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 05:52:24,772::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 05:52:24,772::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 05:52:24,774::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 05:52:24,775::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:52:24,775::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:52:24,776::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:52:24,776::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:52:24,776::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:52:24,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:52:25,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:52:25,202::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:52:25,204::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 05:52:25,206::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 05:52:25,207::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 05:52:25,208::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 05:52:25,209::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:52:25,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:52:25,603::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:52:25,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127943 s, 320 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:52:31,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:52:31,973::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.018302 s, 224 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:52:35,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:52:35,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142585 s, 2.9 MB/s\n'; <rc> = 0
>Thread-15059::WARNING::2013-03-18 05:52:39,313::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:52:39,318::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:52:41,986::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 05:52:41,988::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 05:52:41,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:52:42,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00100745 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:52:42,017::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00100745 s, 2.0 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 05:52:42,017::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 05:52:42,018::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 05:52:42,019::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 05:52:42,020::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:52:42,021::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:52:42,021::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:52:42,022::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:52:42,022::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:52:42,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:52:42,439::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:52:42,444::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:52:42,444::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 05:52:42,447::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 05:52:42,448::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 05:52:42,449::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 05:52:42,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:52:42,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:52:42,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:52:42,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106775 s, 384 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:52:45,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:52:45,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130757 s, 313 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:52:49,325::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:52:49,326::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:52:49,797::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:52:49,803::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:52:49,805::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:52:52,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:52:52,930::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00944114 s, 434 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:52:55,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:52:55,889::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000901764 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:53:02,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:53:02,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147833 s, 277 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:53:05,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:53:05,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783568 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:53:12,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:53:13,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00293378 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:53:16,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:53:16,028::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00197145 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:53:23,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:53:23,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779692 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:53:26,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:53:26,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078453 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:53:33,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:53:33,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133627 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:53:36,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:53:36,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131712 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:53:43,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:53:43,187::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839196 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:53:46,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:53:46,240::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772448 s, 5.3 MB/s\n'; <rc> = 0
>Thread-15060::WARNING::2013-03-18 05:53:49,832::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:53:49,836::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:53:53,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:53:53,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749298 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:53:56,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:53:56,312::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000995104 s, 4.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:53:59,838::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:53:59,839::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:54:00,269::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:54:00,275::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:54:00,277::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:54:03,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:54:03,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175335 s, 234 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:54:06,352::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:54:06,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.023296 s, 176 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:54:13,338::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:54:13,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860502 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:54:16,445::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:54:16,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858834 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:54:23,445::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:54:23,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0070241 s, 583 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:54:26,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:54:26,563::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137467 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:54:33,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:54:33,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785856 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:54:36,603::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:54:36,636::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896235 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:54:43,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:54:43,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138842 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:54:46,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:54:46,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016376 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:54:53,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:54:53,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777063 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:54:56,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:54:56,805::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142097 s, 2.9 MB/s\n'; <rc> = 0
>Thread-15061::WARNING::2013-03-18 05:55:00,301::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:55:00,304::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:55:03,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:55:03,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786192 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:55:06,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:55:06,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010173 s, 4.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:55:10,306::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 05:55:10,307::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:55:10,308::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:55:10,767::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:55:10,773::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:55:10,775::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:55:13,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:55:13,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0283992 s, 144 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:55:16,917::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:55:17,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.050309 s, 81.4 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:55:23,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:55:23,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00366249 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:55:27,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:55:27,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0011816 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:55:33,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:55:34,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00200235 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:55:37,113::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:55:37,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773711 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:55:44,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:55:44,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137721 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:55:47,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:55:47,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00231598 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:55:54,134::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:55:54,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000704916 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:55:57,280::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:55:57,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141925 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:56:04,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:56:04,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185852 s, 220 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:56:07,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:56:07,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000746882 s, 5.5 MB/s\n'; <rc> = 0
>Thread-15062::WARNING::2013-03-18 05:56:10,801::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:56:10,804::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:56:14,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:56:14,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837964 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:56:17,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:56:17,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142547 s, 287 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:56:20,807::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:56:20,808::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:56:21,253::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:56:21,260::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:56:21,261::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:56:24,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:56:24,427::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163755 s, 250 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:56:27,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:56:27,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846617 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:56:34,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:56:34,478::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000914436 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:56:37,606::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:56:37,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154954 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:56:44,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:56:44,553::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0225676 s, 181 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:56:47,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:56:47,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138816 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:56:54,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:56:54,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837031 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:56:57,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:56:57,820::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146035 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:57:04,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:57:04,643::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763687 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:57:07,859::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:57:07,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132287 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:57:14,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:57:14,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864309 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:57:17,941::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:57:17,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141586 s, 2.9 MB/s\n'; <rc> = 0
>Thread-15063::WARNING::2013-03-18 05:57:21,289::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:57:21,294::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:57:24,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:57:24,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865531 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:57:28,022::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 05:57:28,023::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 05:57:28,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:57:28,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000778396 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:57:28,051::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000778396 s, 2.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 05:57:28,051::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 05:57:28,051::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 05:57:28,053::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 05:57:28,054::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:57:28,055::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:57:28,055::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:57:28,056::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:57:28,056::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 05:57:28,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:57:28,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:57:28,478::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 05:57:28,479::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 05:57:28,482::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 05:57:28,483::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 05:57:28,483::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 05:57:28,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:57:28,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:57:28,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:57:28,951::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0080805 s, 507 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:57:31,297::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:57:31,298::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:57:31,827::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:57:31,834::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:57:31,835::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:57:34,732::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:57:34,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119569 s, 343 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:57:38,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:57:39,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013596 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:57:44,783::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 05:57:44,785::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 05:57:44,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:57:44,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00140018 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:57:44,825::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00140018 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 05:57:44,825::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 05:57:44,825::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 05:57:44,827::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 05:57:44,828::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:57:44,829::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:57:44,829::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:57:44,830::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:57:44,830::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 05:57:44,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:57:45,241::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:57:45,246::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:57:45,246::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 05:57:45,249::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 05:57:45,250::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 05:57:45,251::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 05:57:45,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:57:45,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:57:45,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:57:45,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189203 s, 216 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:57:49,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:57:49,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0205518 s, 199 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:57:55,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:57:55,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00299632 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:57:59,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:57:59,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877302 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:58:05,716::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:58:05,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0241496 s, 170 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:58:09,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:58:09,266::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137504 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:58:15,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:58:15,865::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138718 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:58:19,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:58:19,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081466 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:58:25,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:58:25,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835619 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:58:29,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:58:29,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109831 s, 3.7 MB/s\n'; <rc> = 0
>Thread-15064::WARNING::2013-03-18 05:58:31,861::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:58:31,863::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:58:35,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:58:35,936::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789825 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:58:39,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:58:39,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133677 s, 3.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:58:41,874::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:58:41,875::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:58:42,359::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:58:42,366::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:58:42,367::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:58:45,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:58:45,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179335 s, 228 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:58:49,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:58:49,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772717 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:58:55,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:58:56,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829649 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:58:59,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:58:59,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825706 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:59:06,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:59:06,068::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119487 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:59:09,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:59:09,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924667 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:59:16,073::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:59:16,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145946 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:59:19,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:59:19,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000722138 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:59:26,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:59:26,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754741 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:59:29,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:59:29,905::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000915509 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 05:59:36,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:59:36,203::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164572 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:59:39,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:59:39,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872195 s, 4.7 MB/s\n'; <rc> = 0
>Thread-15065::WARNING::2013-03-18 05:59:42,395::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 05:59:42,401::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 05:59:46,211::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:59:46,239::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804637 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 05:59:50,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 05:59:50,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008099 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 05:59:52,403::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 05:59:52,404::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 05:59:52,866::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 05:59:52,872::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 05:59:52,874::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 05:59:56,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 05:59:56,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147606 s, 277 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:00:00,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:00:00,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898634 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:00:06,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:00:06,352::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134937 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:00:10,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:00:10,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865607 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:00:16,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:00:16,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112038 s, 3.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:00:20,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:00:20,266::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795025 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:00:26,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:00:26,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000988228 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:00:30,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:00:30,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162143 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:00:36,436::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:00:36,470::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878194 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:00:40,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:00:40,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135214 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:00:46,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:00:46,517::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00514948 s, 795 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:00:50,482::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:00:50,513::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775766 s, 5.3 MB/s\n'; <rc> = 0
>Thread-15066::WARNING::2013-03-18 06:00:52,911::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:00:52,917::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 06:00:56,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:00:56,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000930568 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:01:00,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:01:00,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787947 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:01:02,919::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 06:01:02,920::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:01:02,921::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:01:03,401::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:01:03,408::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:01:03,409::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:01:06,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:01:06,631::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151568 s, 270 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:01:10,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:01:10,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854661 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:01:16,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:01:16,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787128 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:01:20,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:01:20,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137372 s, 298 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:01:26,670::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:01:26,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130674 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:01:30,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:01:30,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000877268 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:01:36,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:01:36,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000947493 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:01:40,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:01:40,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820544 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:01:46,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:01:46,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140686 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:01:50,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:01:50,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142576 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:01:56,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:01:56,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125476 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:02:01,006::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:02:01,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109583 s, 3.7 MB/s\n'; <rc> = 0
>Thread-15067::WARNING::2013-03-18 06:02:03,436::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:02:03,440::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 06:02:06,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:02:06,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809999 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:02:11,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:02:11,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000846236 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:02:13,442::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:02:13,443::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:02:13,830::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:02:13,836::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:02:13,838::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:02:16,893::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:02:16,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00660274 s, 620 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:02:21,209::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:02:21,238::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839781 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:02:26,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:02:26,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00215548 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:02:31,277::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 06:02:31,278::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 06:02:31,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:02:31,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00120971 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:02:31,323::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00120971 s, 1.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 06:02:31,323::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 06:02:31,323::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 06:02:31,325::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 06:02:31,326::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:02:31,327::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:02:31,327::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:02:31,327::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:02:31,328::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:02:31,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:02:31,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:02:31,730::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:02:31,731::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 06:02:31,734::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 06:02:31,735::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 06:02:31,735::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 06:02:31,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:02:32,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:02:32,095::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:02:32,128::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00497219 s, 824 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:02:36,986::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:02:37,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0302009 s, 136 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:02:42,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:02:42,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774491 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:02:47,066::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 06:02:47,067::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 06:02:47,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:02:47,107::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00148878 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:02:47,108::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00148878 s, 1.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 06:02:47,109::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 06:02:47,109::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 06:02:47,111::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 06:02:47,112::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:02:47,113::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:02:47,113::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:02:47,113::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:02:47,114::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:02:47,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:02:47,571::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:02:47,575::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:02:47,576::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 06:02:47,578::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 06:02:47,579::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 06:02:47,580::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 06:02:47,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:02:47,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:02:47,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:02:48,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0362644 s, 113 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:02:52,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:02:52,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853509 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:02:58,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:02:58,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130489 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:03:02,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:03:02,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158445 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:03:08,048::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:03:08,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163897 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:03:12,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:03:12,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800783 s, 5.1 MB/s\n'; <rc> = 0
>Thread-15068::WARNING::2013-03-18 06:03:13,864::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:03:13,869::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 06:03:18,096::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:03:18,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853695 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:03:23,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:03:23,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844671 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:03:23,871::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:03:23,872::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:03:24,381::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:03:24,388::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:03:24,389::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:03:28,132::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:03:28,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00405894 s, 1.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:03:33,079::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:03:33,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00214351 s, 1.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:03:38,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:03:38,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00065393 s, 6.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:03:43,145::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:03:43,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151815 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:03:48,209::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:03:48,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000996646 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:03:53,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:03:53,272::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133672 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:03:58,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:03:58,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157225 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:04:03,311::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:04:03,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000907988 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:04:08,305::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:04:08,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878003 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:04:13,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:04:13,407::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897071 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:04:18,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:04:18,382::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117936 s, 3.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:04:23,681::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:04:23,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0032032 s, 1.3 MB/s\n'; <rc> = 0
>Thread-15069::WARNING::2013-03-18 06:04:24,420::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:04:24,432::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 06:04:28,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:04:28,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909012 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:04:33,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:04:33,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882861 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:04:34,443::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:04:34,445::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:04:34,858::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:04:34,864::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:04:34,865::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:04:38,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:04:38,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136916 s, 299 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:04:44,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:04:44,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110941 s, 369 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:04:48,482::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:04:48,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150288 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:04:54,092::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:04:54,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798581 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:04:58,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:04:58,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132459 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:05:04,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:05:04,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164473 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:05:08,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:05:08,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014413 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:05:14,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:05:14,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155723 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:05:18,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:05:18,648::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144682 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:05:24,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:05:24,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014595 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:05:28,653::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:05:28,692::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000908108 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:05:34,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:05:34,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816453 s, 5.0 MB/s\n'; <rc> = 0
>Thread-15070::WARNING::2013-03-18 06:05:34,893::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:05:34,898::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 06:05:38,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:05:38,727::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829994 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:05:44,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:05:44,755::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00945975 s, 433 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:05:44,900::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:05:44,902::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:05:45,341::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:05:45,348::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:05:45,349::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:05:48,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:05:48,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192357 s, 213 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:05:54,810::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:05:54,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079694 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:05:58,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:05:58,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140421 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:06:04,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:06:04,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858617 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:06:08,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:06:08,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786839 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:06:14,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:06:15,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129718 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:06:18,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:06:18,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135003 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:06:25,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:06:25,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0158468 s, 258 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:06:28,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:06:28,952::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103801 s, 3.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:06:35,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:06:35,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157956 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:06:38,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:06:38,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780447 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:06:45,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:06:45,287::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149556 s, 2.7 MB/s\n'; <rc> = 0
>Thread-15071::WARNING::2013-03-18 06:06:45,376::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:06:45,395::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 06:06:48,994::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:06:49,021::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000819487 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:06:55,409::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 06:06:55,411::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:06:55,412::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:06:55,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:06:55,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132383 s, 309 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:06:55,876::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:06:55,881::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:06:55,883::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:06:59,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:06:59,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173519 s, 236 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:07:05,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:07:05,593::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145172 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:07:09,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:07:09,105::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091148 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:07:15,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:07:15,657::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792969 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:07:19,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:07:19,140::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807824 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:07:25,697::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:07:25,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135867 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:07:29,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:07:29,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850227 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:07:35,779::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 06:07:35,781::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 06:07:35,782::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:07:35,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00107453 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:07:35,812::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00107453 s, 1.9 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 06:07:35,812::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 06:07:35,812::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 06:07:35,814::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 06:07:35,815::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:07:35,816::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:07:35,816::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:07:35,817::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:07:35,817::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:07:35,818::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:07:36,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:07:36,257::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:07:36,258::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 06:07:36,260::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 06:07:36,262::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 06:07:36,262::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 06:07:36,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:07:36,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:07:36,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:07:36,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00770522 s, 532 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:07:39,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:07:39,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138675 s, 295 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:07:46,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:07:46,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149535 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:07:49,248::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 06:07:49,249::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 06:07:49,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:07:49,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000992715 s, 2.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:07:49,287::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000992715 s, 2.1 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 06:07:49,288::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 06:07:49,288::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 06:07:49,290::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 06:07:49,292::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:07:49,292::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:07:49,293::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:07:49,293::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:07:49,294::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:07:49,295::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:07:49,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:07:49,726::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:07:49,729::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 06:07:49,731::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 06:07:49,732::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 06:07:49,733::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 06:07:49,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:07:50,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:07:50,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:07:50,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0041741 s, 981 kB/s\n'; <rc> = 0
>Thread-15072::WARNING::2013-03-18 06:07:55,906::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:07:55,912::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 06:07:56,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:07:56,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014709 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:08:00,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:08:00,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753818 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:08:05,915::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:08:05,916::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:08:06,316::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:08:06,323::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:08:06,324::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:08:06,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:08:06,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0150782 s, 272 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:08:10,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:08:10,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176772 s, 232 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:08:16,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:08:17,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839019 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:08:20,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:08:20,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00992593 s, 413 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:08:27,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:08:27,096::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143953 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:08:30,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:08:30,310::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776497 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:08:37,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:08:37,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008837 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:08:40,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:08:40,355::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147746 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:08:47,209::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:08:47,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138402 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:08:50,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:08:50,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137381 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:08:57,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:08:57,319::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770797 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:09:00,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:09:00,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156019 s, 2.6 MB/s\n'; <rc> = 0
>Thread-15073::WARNING::2013-03-18 06:09:06,352::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:09:06,356::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 06:09:07,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:09:07,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010162 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:09:10,445::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:09:10,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867363 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:09:16,359::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:09:16,360::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:09:16,759::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:09:16,765::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:09:16,769::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:09:17,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:09:17,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145606 s, 281 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:09:20,481::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:09:20,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0153747 s, 266 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:09:27,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:09:27,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074139 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:09:30,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:09:30,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0058282 s, 703 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:09:37,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:09:37,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108125 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:09:40,585::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:09:40,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783483 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:09:47,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:09:47,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000768031 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:09:50,623::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:09:50,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00752731 s, 544 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:09:57,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:09:57,786::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131316 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:10:00,675::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:10:00,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899314 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:10:07,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:10:07,874::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123896 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:10:10,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:10:10,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00708558 s, 578 kB/s\n'; <rc> = 0
>Thread-15074::WARNING::2013-03-18 06:10:16,798::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:10:16,802::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 06:10:17,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:10:17,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107155 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:10:20,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:10:20,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086036 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:10:26,804::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:10:26,805::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:10:27,210::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:10:27,217::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:10:27,218::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:10:27,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:10:28,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00918751 s, 446 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:10:30,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:10:30,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0176486 s, 232 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:10:38,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:10:38,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145437 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:10:40,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:10:40,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.036201 s, 113 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:10:48,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:10:48,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109028 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:10:50,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:10:50,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896574 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:10:58,229::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:10:58,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811863 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:11:00,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:11:00,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829207 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:11:08,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:11:08,347::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001218 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:11:11,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:11:11,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014485 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:11:18,387::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:11:18,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141863 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:11:21,048::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:11:21,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00227638 s, 1.8 MB/s\n'; <rc> = 0
>Thread-15075::WARNING::2013-03-18 06:11:27,248::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:11:27,253::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 06:11:28,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:11:28,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854732 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:11:31,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:11:31,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086781 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:11:37,255::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:11:37,256::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:11:37,715::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:11:37,722::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:11:37,723::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:11:38,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:11:38,611::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100046 s, 409 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:11:41,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:11:41,179::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0116429 s, 352 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:11:48,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:11:48,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135035 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:11:51,185::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:11:51,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000975736 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:11:58,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:11:58,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763054 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:12:01,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:12:01,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143523 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:12:08,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:12:08,836::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897544 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:12:11,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:12:11,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000707129 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:12:18,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:12:18,921::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137402 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:12:21,303::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:12:21,347::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015429 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:12:28,961::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:12:28,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150321 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:12:31,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:12:31,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878179 s, 4.7 MB/s\n'; <rc> = 0
>Thread-15076::WARNING::2013-03-18 06:12:37,747::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:12:37,751::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 06:12:39,038::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 06:12:39,039::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 06:12:39,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:12:39,065::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000902005 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:12:39,067::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000902005 s, 2.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 06:12:39,068::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 06:12:39,068::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 06:12:39,070::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 06:12:39,072::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:12:39,073::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:12:39,073::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:12:39,074::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:12:39,074::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:12:39,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:12:39,513::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:12:39,517::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:12:39,518::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 06:12:39,521::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 06:12:39,522::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 06:12:39,522::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 06:12:39,524::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:12:39,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:12:39,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:12:39,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00688878 s, 595 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:12:41,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:12:41,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149166 s, 275 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:12:47,753::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 06:12:47,754::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:12:47,755::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:12:48,212::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:12:48,219::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:12:48,220::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:12:49,979::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:12:50,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113727 s, 360 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:12:51,447::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 06:12:51,448::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 06:12:51,448::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:12:51,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.018409 s, 111 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:12:51,509::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.018409 s, 111 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 06:12:51,510::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 06:12:51,510::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 06:12:51,512::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 06:12:51,513::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:12:51,514::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:12:51,514::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:12:51,515::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:12:51,515::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:12:51,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:12:51,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:12:51,929::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:12:51,930::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 06:12:51,932::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 06:12:51,933::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 06:12:51,934::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 06:12:51,935::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:12:52,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:12:52,302::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:12:52,347::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126722 s, 323 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:13:00,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:13:00,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00234017 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:13:02,352::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:13:02,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899345 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:13:10,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:13:10,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843159 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:13:12,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:13:12,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152827 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:13:20,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:13:20,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818582 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:13:22,432::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:13:22,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141711 s, 289 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:13:30,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:13:30,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132822 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:13:32,482::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:13:32,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148033 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:13:40,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:13:40,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133007 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:13:42,530::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:13:42,569::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000906371 s, 4.5 MB/s\n'; <rc> = 0
>Thread-15077::WARNING::2013-03-18 06:13:48,244::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:13:48,250::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 06:13:50,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:13:50,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879758 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:13:52,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:13:52,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000945887 s, 4.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:13:58,253::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:13:58,254::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:13:58,666::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:13:58,673::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:13:58,674::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:14:00,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:14:00,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0231848 s, 177 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:14:02,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:14:02,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145364 s, 282 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:14:10,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:14:10,641::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764809 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:14:12,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:14:12,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883029 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:14:20,680::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:14:20,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000896028 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:14:22,694::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:14:22,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088437 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:14:30,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:14:30,785::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814263 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:14:32,726::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:14:32,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000618009 s, 6.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:14:40,830::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:14:40,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131497 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:14:42,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:14:42,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147854 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:14:50,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:14:50,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00204725 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:14:52,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:14:52,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144765 s, 2.8 MB/s\n'; <rc> = 0
>Thread-15078::WARNING::2013-03-18 06:14:58,701::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:14:58,706::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 06:15:01,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:15:01,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000843841 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:15:02,860::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:15:02,886::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858439 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:15:08,710::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:15:08,711::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:15:09,177::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:15:09,184::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:15:09,185::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:15:11,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:15:11,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00859403 s, 477 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:15:12,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:15:12,941::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165895 s, 247 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:15:21,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:15:21,191::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890239 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:15:22,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:15:22,977::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000873169 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:15:31,230::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:15:31,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0223075 s, 184 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:15:32,982::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:15:33,025::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014434 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:15:41,324::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:15:41,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000946559 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:15:43,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:15:43,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154312 s, 265 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:15:51,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:15:51,441::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150503 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:15:53,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:15:53,127::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755603 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:16:01,480::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:16:01,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817284 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:16:03,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:16:03,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102387 s, 400 kB/s\n'; <rc> = 0
>Thread-15079::WARNING::2013-03-18 06:16:09,218::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:16:09,222::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 06:16:11,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:16:11,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818516 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:16:13,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:16:13,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000722894 s, 5.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:16:19,227::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:16:19,228::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:16:19,749::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:16:19,755::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:16:19,756::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:16:21,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:16:21,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00744543 s, 550 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:16:23,264::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:16:23,329::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165198 s, 248 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:16:31,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:16:31,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796517 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:16:33,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:16:33,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923546 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:16:41,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:16:41,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855703 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:16:43,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:16:43,414::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138815 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:16:51,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:16:51,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125004 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:16:53,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:16:53,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140627 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:17:01,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:17:01,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138266 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:17:03,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:17:03,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088308 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:17:12,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:17:12,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135412 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:17:13,503::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:17:13,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010118 s, 4.0 MB/s\n'; <rc> = 0
>Thread-15080::WARNING::2013-03-18 06:17:19,782::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:17:19,786::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 06:17:22,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:17:22,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893869 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:17:23,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:17:23,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000949473 s, 4.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:17:29,788::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:17:29,789::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:17:30,202::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:17:30,209::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:17:30,210::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:17:32,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:17:32,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115489 s, 355 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:17:33,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:17:33,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117008 s, 350 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:17:42,241::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 06:17:42,243::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 06:17:42,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:17:42,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0137061 s, 149 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:17:42,286::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0137061 s, 149 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 06:17:42,287::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 06:17:42,287::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 06:17:42,289::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 06:17:42,291::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:17:42,292::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:17:42,295::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:17:42,295::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:17:42,296::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:17:42,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:17:42,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:17:42,720::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:17:42,721::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 06:17:42,725::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 06:17:42,726::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 06:17:42,726::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 06:17:42,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:17:43,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:17:43,146::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:17:43,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00947022 s, 433 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:17:43,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:17:43,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136577 s, 300 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:17:53,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:17:53,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086377 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:17:53,693::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 06:17:53,699::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 06:17:53,700::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:17:53,736::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00185435 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:17:53,742::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00185435 s, 1.1 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 06:17:53,742::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 06:17:53,743::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 06:17:53,744::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 06:17:53,745::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:17:53,746::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:17:53,746::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:17:53,746::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:17:53,747::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:17:53,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:17:54,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:17:54,185::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:17:54,185::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 06:17:54,188::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 06:17:54,189::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 06:17:54,190::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 06:17:54,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:17:54,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:17:54,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:17:54,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186273 s, 220 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:18:03,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:18:03,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145203 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:18:04,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:18:04,636::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00647567 s, 633 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:18:13,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:18:13,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138136 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:18:14,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:18:14,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133571 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:18:23,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:18:23,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108711 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:18:24,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:18:24,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0192252 s, 213 kB/s\n'; <rc> = 0
>Thread-15081::WARNING::2013-03-18 06:18:30,235::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:18:30,238::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 06:18:33,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:18:33,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828103 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:18:34,762::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:18:34,791::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814607 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:18:40,250::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 06:18:40,251::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:18:40,252::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:18:40,669::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:18:40,676::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:18:40,677::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:18:43,629::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:18:43,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0165919 s, 247 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:18:44,797::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:18:44,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164544 s, 249 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:18:53,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:18:53,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797389 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:18:54,853::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:18:54,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000974168 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:19:03,801::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:19:03,841::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146193 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:19:04,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:19:04,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147454 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:19:13,881::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:19:13,913::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132329 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:19:14,938::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:19:14,978::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148201 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:19:23,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:19:23,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104766 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:19:24,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:19:25,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144804 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:19:34,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:19:34,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884404 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:19:35,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:19:35,072::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147196 s, 2.8 MB/s\n'; <rc> = 0
>Thread-15082::WARNING::2013-03-18 06:19:40,706::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:19:40,709::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 06:19:44,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:19:44,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854805 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:19:45,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:19:45,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833939 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:19:50,721::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:19:50,722::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:19:51,177::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:19:51,184::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:19:51,185::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:19:54,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:19:54,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103033 s, 398 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:19:55,115::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:19:55,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016988 s, 241 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:20:04,269::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:20:04,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105726 s, 387 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:20:05,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:20:05,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141711 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:20:14,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:20:14,380::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000716219 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:20:15,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:20:15,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136397 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:20:24,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:20:24,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00363711 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:20:25,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:20:25,306::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126573 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:20:34,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:20:34,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146702 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:20:35,312::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:20:35,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00363513 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:20:44,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:20:44,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784268 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:20:45,358::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:20:45,393::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000738129 s, 5.5 MB/s\n'; <rc> = 0
>Thread-15083::WARNING::2013-03-18 06:20:51,210::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:20:51,213::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 06:20:54,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:20:54,712::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144371 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:20:55,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:20:55,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000908626 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:21:01,221::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:21:01,222::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:21:01,657::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:21:01,664::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:21:01,665::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:21:04,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:21:04,791::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0128746 s, 318 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:21:05,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:21:05,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127658 s, 321 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:21:14,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:21:14,857::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805653 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:21:15,494::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:21:15,523::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765313 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:21:24,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:21:24,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802688 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:21:25,529::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:21:25,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000937472 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:21:34,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:21:35,009::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786761 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:21:35,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:21:35,611::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150657 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:21:45,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:21:45,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141828 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:21:45,617::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:21:45,671::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153314 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:21:55,159::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:21:55,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091804 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:21:55,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:21:55,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000643051 s, 6.4 MB/s\n'; <rc> = 0
>Thread-15084::WARNING::2013-03-18 06:22:01,694::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:22:01,698::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 06:22:05,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:22:05,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806056 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:22:05,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:22:05,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865711 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:22:11,702::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:22:11,703::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:22:12,109::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:22:12,116::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:22:12,117::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:22:15,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:22:15,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186483 s, 220 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:22:15,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:22:15,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131572 s, 311 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:22:25,405::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:22:25,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143323 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:22:25,803::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:22:25,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085609 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:22:35,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:22:35,524::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162903 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:22:35,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:22:35,888::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181966 s, 225 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:22:45,563::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 06:22:45,564::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 06:22:45,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:22:45,591::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000740938 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:22:45,593::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000740938 s, 2.8 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 06:22:45,593::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 06:22:45,594::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 06:22:45,595::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 06:22:45,596::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:22:45,597::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:22:45,597::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:22:45,598::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:22:45,598::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:22:45,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:22:45,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:22:45,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000627688 s, 6.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:22:46,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:22:46,062::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:22:46,062::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 06:22:46,066::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 06:22:46,067::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 06:22:46,067::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 06:22:46,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:22:46,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:22:46,490::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:22:46,532::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137908 s, 297 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:22:55,934::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 06:22:55,935::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 06:22:55,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:22:56,000::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.021409 s, 95.7 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:22:56,002::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.021409 s, 95.7 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 06:22:56,002::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 06:22:56,003::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 06:22:56,004::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 06:22:56,005::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:22:56,006::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:22:56,007::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:22:56,007::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:22:56,007::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:22:56,013::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:22:56,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:22:56,443::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:22:56,444::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 06:22:56,446::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 06:22:56,447::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 06:22:56,448::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 06:22:56,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:22:56,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:22:56,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141143 s, 290 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:22:56,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:22:56,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:22:56,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0211345 s, 194 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:23:06,688::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:23:06,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148527 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:23:06,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:23:06,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0018534 s, 2.2 MB/s\n'; <rc> = 0
>Thread-15085::WARNING::2013-03-18 06:23:12,155::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:23:12,161::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 06:23:16,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:23:16,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0440164 s, 93.1 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:23:17,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:23:17,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141257 s, 290 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:23:22,164::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:23:22,165::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:23:22,605::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:23:22,611::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:23:22,613::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:23:26,877::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:23:26,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0105337 s, 389 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:23:27,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:23:27,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155721 s, 263 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:23:36,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:23:37,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00209186 s, 2.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:23:37,117::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:23:37,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173935 s, 235 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:23:47,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:23:47,102::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162332 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:23:47,171::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:23:47,207::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00196098 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:23:57,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:23:57,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000876789 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:23:57,218::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:23:57,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136123 s, 301 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:24:07,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:24:07,285::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:24:07,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000939801 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:24:07,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815103 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:24:17,378::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:24:17,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:24:17,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010643 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:24:17,510::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175595 s, 2.3 MB/s\n'; <rc> = 0
>Thread-15086::WARNING::2013-03-18 06:24:22,645::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:24:22,648::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 06:24:27,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:24:27,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100684 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:24:27,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:24:27,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759396 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:24:32,649::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 06:24:32,650::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:24:32,651::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:24:33,093::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:24:33,100::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:24:33,101::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:24:37,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:24:37,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:24:37,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014927 s, 274 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:24:37,697::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142446 s, 288 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:24:47,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:24:47,729::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831084 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:24:47,736::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:24:47,762::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084359 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:24:57,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:24:57,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858151 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:24:57,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:24:57,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000868369 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:25:07,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:25:07,839::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0281521 s, 145 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:25:07,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:25:07,907::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000928308 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:25:17,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:25:17,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000893558 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:25:17,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:25:17,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804663 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:25:27,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:25:27,923::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000960835 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:25:28,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:25:28,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000944808 s, 4.3 MB/s\n'; <rc> = 0
>Thread-15087::WARNING::2013-03-18 06:25:33,136::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:25:33,143::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 06:25:37,929::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:25:37,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936726 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:25:38,110::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:25:38,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149107 s, 2.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:25:43,146::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:25:43,154::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:25:43,623::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:25:43,629::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:25:43,630::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:25:47,964::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:25:48,017::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142438 s, 288 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:25:48,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:25:48,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147434 s, 278 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:25:58,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:25:58,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163525 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:25:58,283::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:25:58,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878531 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:26:08,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:26:08,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164458 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:26:08,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:26:08,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898872 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:26:18,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:26:18,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129538 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:26:18,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:26:18,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103917 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:26:28,170::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:26:28,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131528 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:26:28,492::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:26:28,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136729 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:26:38,214::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:26:38,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144762 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:26:38,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:26:38,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134514 s, 3.0 MB/s\n'; <rc> = 0
>Thread-15088::WARNING::2013-03-18 06:26:43,657::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:26:43,660::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 06:26:48,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:26:48,287::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824623 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:26:48,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:26:48,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179713 s, 228 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:26:53,662::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:26:53,663::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:26:54,194::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:26:54,201::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:26:54,202::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:26:58,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:26:58,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102912 s, 398 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:26:58,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:26:58,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134345 s, 305 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:27:08,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:27:08,369::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806268 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:27:08,831::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:27:08,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139442 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:27:18,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:27:18,416::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156888 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:27:18,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:27:18,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164968 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:27:28,421::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:27:28,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160388 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:27:28,996::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:27:29,032::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000735556 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:27:38,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:27:38,522::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174172 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:27:39,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:27:39,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162621 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:27:48,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:27:48,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161673 s, 253 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:27:49,149::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 06:27:49,151::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 06:27:49,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:27:49,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0014601 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:27:49,199::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0014601 s, 1.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 06:27:49,200::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 06:27:49,200::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 06:27:49,202::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 06:27:49,203::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:27:49,204::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:27:49,204::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:27:49,204::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:27:49,205::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:27:49,206::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:27:49,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:27:49,661::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:27:49,668::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 06:27:49,671::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 06:27:49,672::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 06:27:49,672::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 06:27:49,674::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:27:50,040::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:27:50,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:27:50,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127323 s, 322 kB/s\n'; <rc> = 0
>Thread-15089::WARNING::2013-03-18 06:27:54,236::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:27:54,241::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 06:27:58,584::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 06:27:58,586::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 06:27:58,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:27:58,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000802057 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:27:58,615::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000802057 s, 2.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 06:27:58,616::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 06:27:58,616::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 06:27:58,617::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 06:27:58,618::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:27:58,619::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:27:58,619::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:27:58,620::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:27:58,620::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:27:58,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:27:59,054::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:27:59,058::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:27:59,059::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 06:27:59,062::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 06:27:59,064::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 06:27:59,064::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 06:27:59,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:27:59,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:27:59,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:27:59,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125828 s, 326 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:28:00,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:28:00,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0074361 s, 551 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:28:04,243::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:28:04,244::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:28:04,732::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:28:04,739::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:28:04,740::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:28:09,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:28:09,511::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107585 s, 381 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:28:10,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:28:10,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142605 s, 287 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:28:19,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:28:19,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142285 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:28:20,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:28:20,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00358108 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:28:29,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:28:29,608::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100965 s, 406 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:28:30,390::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:28:30,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00611982 s, 669 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:28:39,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:28:39,644::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799499 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:28:40,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:28:40,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137709 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:28:49,650::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:28:49,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140812 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:28:50,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:28:50,601::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154262 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:28:59,695::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:28:59,738::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156456 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:29:00,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:29:00,689::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015257 s, 2.7 MB/s\n'; <rc> = 0
>Thread-15090::WARNING::2013-03-18 06:29:04,766::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:29:04,781::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 06:29:09,745::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:29:09,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815045 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:29:10,730::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:29:10,757::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000935842 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:29:14,794::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:29:14,795::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:29:15,262::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:29:15,268::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:29:15,270::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:29:19,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:29:19,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.011468 s, 357 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:29:20,796::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:29:20,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000856212 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:29:29,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:29:29,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170468 s, 240 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:29:30,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:29:30,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000988207 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:29:39,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:29:39,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082027 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:29:40,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:29:40,965::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102138 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:29:49,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:29:49,969::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0012952 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:29:51,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:29:51,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789959 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:29:59,974::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:30:00,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146342 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:30:01,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:30:01,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0330443 s, 124 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:30:10,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:30:10,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163523 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:30:11,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:30:11,260::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000885308 s, 4.6 MB/s\n'; <rc> = 0
>Thread-15091::WARNING::2013-03-18 06:30:15,308::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:30:15,315::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 06:30:20,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:30:20,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000941227 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:30:21,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:30:21,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112588 s, 3.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:30:25,317::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 06:30:25,318::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:30:25,319::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:30:25,855::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:30:25,861::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:30:25,863::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:30:30,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:30:30,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.012762 s, 321 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:30:31,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:30:31,428::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00734531 s, 558 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:30:40,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:30:40,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762103 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:30:41,467::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:30:41,512::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015128 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:30:50,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:30:50,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131409 s, 312 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:30:51,553::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:30:51,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150117 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:31:00,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:31:00,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144974 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:31:01,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:31:01,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000990207 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:31:10,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:31:10,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118575 s, 345 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:31:11,702::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:31:11,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825198 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:31:20,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:31:20,367::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008109 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:31:21,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:31:21,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159849 s, 2.6 MB/s\n'; <rc> = 0
>Thread-15092::WARNING::2013-03-18 06:31:25,890::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:31:25,894::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 06:31:30,373::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:31:30,408::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152523 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:31:31,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:31:31,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113334 s, 361 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:31:35,896::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:31:35,897::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:31:36,322::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:31:36,329::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:31:36,330::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:31:40,415::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:31:40,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160827 s, 255 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:31:41,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:31:42,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143785 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:31:50,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:31:50,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919468 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:31:52,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:31:52,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815985 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:32:00,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:32:00,557::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138316 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:32:02,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:32:02,196::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133959 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:32:10,562::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:32:10,622::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160322 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:32:12,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:32:12,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777292 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:32:20,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:32:20,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015265 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:32:22,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:32:22,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151223 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:32:30,678::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:32:30,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156508 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:32:32,392::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:32:32,426::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136696 s, 3.0 MB/s\n'; <rc> = 0
>Thread-15093::WARNING::2013-03-18 06:32:36,360::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:32:36,371::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 06:32:40,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:32:40,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000853658 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:32:42,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:32:42,500::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810824 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:32:46,374::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:32:46,375::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:32:46,848::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:32:46,854::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:32:46,856::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:32:50,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:32:50,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122935 s, 333 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:32:52,544::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 06:32:52,546::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 06:32:52,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:32:52,602::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00156248 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:32:52,603::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00156248 s, 1.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 06:32:52,604::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 06:32:52,604::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 06:32:52,605::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 06:32:52,606::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:32:52,607::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:32:52,607::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:32:52,608::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:32:52,608::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:32:52,609::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:32:52,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:32:52,998::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:32:52,999::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 06:32:53,002::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 06:32:53,003::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 06:32:53,003::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 06:32:53,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:32:53,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:32:53,356::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:32:53,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00724937 s, 565 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:33:00,805::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 06:33:00,806::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 06:33:00,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:33:00,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0118452 s, 173 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:33:00,848::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0118452 s, 173 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 06:33:00,848::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 06:33:00,849::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 06:33:00,851::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 06:33:00,852::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:33:00,853::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:33:00,853::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:33:00,854::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:33:00,854::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:33:00,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:33:01,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:33:01,289::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:33:01,290::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 06:33:01,293::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 06:33:01,294::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 06:33:01,295::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 06:33:01,296::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:33:01,657::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:33:01,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:33:01,705::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170587 s, 240 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:33:03,438::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:33:03,478::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136989 s, 299 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:33:11,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:33:11,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00718175 s, 570 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:33:13,522::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:33:13,547::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000755885 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:33:21,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:33:21,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813291 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:33:23,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:33:23,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140058 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:33:31,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:33:31,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101698 s, 403 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:33:33,671::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:33:33,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140943 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:33:41,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:33:41,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016987 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:33:43,761::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:33:43,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00444612 s, 921 kB/s\n'; <rc> = 0
>Thread-15094::WARNING::2013-03-18 06:33:46,882::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:33:46,886::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 06:33:51,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:33:51,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159842 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:33:53,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:33:53,863::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814749 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:33:56,888::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:33:56,889::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:33:57,366::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:33:57,373::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:33:57,374::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:34:01,936::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:34:02,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0216814 s, 189 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:34:03,905::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:34:03,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00292192 s, 1.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:34:12,009::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:34:12,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0026378 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:34:13,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:34:14,001::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789223 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:34:22,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:34:22,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827495 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:34:24,047::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:34:24,091::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117709 s, 348 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:34:32,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:34:32,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142238 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:34:34,131::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:34:34,162::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133408 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:34:42,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:34:42,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899059 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:34:44,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:34:44,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00925134 s, 443 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:34:52,178::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:34:52,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151075 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:34:54,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:34:54,345::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144495 s, 2.8 MB/s\n'; <rc> = 0
>Thread-15095::WARNING::2013-03-18 06:34:57,404::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:34:57,408::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 06:35:02,221::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:35:02,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076737 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:35:04,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:35:04,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852953 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:35:07,410::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:35:07,411::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:35:07,856::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:35:07,863::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:35:07,864::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:35:12,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:35:12,307::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177433 s, 231 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:35:14,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:35:14,484::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769733 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:35:22,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:35:22,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00947595 s, 432 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:35:24,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:35:24,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014893 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:35:32,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:35:32,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077063 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:35:34,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:35:34,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104138 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:35:42,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:35:42,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777247 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:35:44,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:35:44,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00234315 s, 1.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:35:52,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:35:52,454::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884794 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:35:54,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:35:54,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0121299 s, 338 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:36:02,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:36:02,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132022 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:36:04,862::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:36:04,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157623 s, 2.6 MB/s\n'; <rc> = 0
>Thread-15096::WARNING::2013-03-18 06:36:07,897::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:36:07,901::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 06:36:12,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:36:12,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793671 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:36:14,949::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:36:14,976::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850675 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:36:17,903::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 06:36:17,904::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:36:17,905::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:36:18,416::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:36:18,422::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:36:18,424::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:36:22,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:36:22,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016772 s, 244 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:36:25,016::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:36:25,046::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000928279 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:36:32,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:36:32,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000734206 s, 5.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:36:35,087::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:36:35,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161389 s, 254 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:36:42,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:36:42,676::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123924 s, 331 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:36:45,175::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:36:45,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863581 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:36:52,681::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:36:52,709::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143453 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:36:55,247::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:36:55,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146465 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:37:02,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:37:02,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167723 s, 244 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:37:05,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:37:05,378::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167652 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:37:12,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:37:12,824::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158669 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:37:15,418::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:37:15,461::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164651 s, 2.5 MB/s\n'; <rc> = 0
>Thread-15097::WARNING::2013-03-18 06:37:18,457::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:37:18,461::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 06:37:22,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:37:22,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152518 s, 269 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:37:25,502::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:37:25,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859291 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:37:28,462::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:37:28,463::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:37:28,917::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:37:28,923::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:37:28,925::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:37:32,887::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:37:32,927::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100841 s, 406 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:37:35,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:37:35,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847147 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:37:42,933::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:37:42,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155166 s, 264 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:37:45,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:37:45,667::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000934188 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:37:52,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:37:53,008::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797264 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:37:55,720::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 06:37:55,721::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 06:37:55,722::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:37:55,798::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00156331 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:37:55,800::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00156331 s, 1.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 06:37:55,800::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 06:37:55,800::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 06:37:55,802::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 06:37:55,803::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:37:55,804::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:37:55,804::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:37:55,804::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:37:55,805::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:37:55,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:37:56,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:37:56,249::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:37:56,250::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 06:37:56,253::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 06:37:56,254::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 06:37:56,254::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 06:37:56,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:37:56,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:37:56,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:37:56,674::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00500234 s, 819 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:38:03,012::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 06:38:03,013::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 06:38:03,014::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:38:03,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0135607 s, 151 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:38:03,056::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0135607 s, 151 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 06:38:03,056::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 06:38:03,057::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 06:38:03,058::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 06:38:03,059::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:38:03,060::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:38:03,060::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:38:03,061::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:38:03,061::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:38:03,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:38:03,487::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:38:03,491::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:38:03,492::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 06:38:03,494::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 06:38:03,496::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 06:38:03,496::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 06:38:03,498::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:38:03,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:38:03,924::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:38:03,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106506 s, 385 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:38:06,714::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:38:06,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137473 s, 298 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:38:13,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:38:14,007::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158409 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:38:16,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:38:16,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138478 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:38:24,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:38:24,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107278 s, 382 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:38:26,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:38:26,936::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001024 s, 4.0 MB/s\n'; <rc> = 0
>Thread-15098::WARNING::2013-03-18 06:38:28,954::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:38:28,960::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 06:38:34,069::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:38:34,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0244606 s, 167 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:38:36,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:38:37,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932299 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:38:38,962::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:38:38,963::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:38:39,471::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:38:39,478::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:38:39,479::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:38:44,142::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:38:44,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0495515 s, 82.7 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:38:47,059::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:38:47,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0194257 s, 211 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:38:54,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:38:54,268::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803564 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:38:57,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:38:57,183::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000715503 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:39:04,273::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:39:04,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103018 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:39:07,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:39:07,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788501 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:39:14,313::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:39:14,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146707 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:39:17,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:39:17,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144907 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:39:24,362::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:39:24,418::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126738 s, 323 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:39:27,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:39:27,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135816 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:39:34,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:39:34,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158242 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:39:37,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:39:37,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129666 s, 3.2 MB/s\n'; <rc> = 0
>Thread-15099::WARNING::2013-03-18 06:39:39,509::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:39:39,517::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 06:39:44,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:39:44,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136517 s, 300 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:39:47,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:39:47,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836638 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:39:49,519::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:39:49,520::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:39:49,989::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:39:49,995::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:39:49,996::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:39:54,539::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:39:54,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0133155 s, 308 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:39:57,618::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:39:57,649::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000866765 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:40:04,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:40:04,626::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00937226 s, 437 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:40:07,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:40:07,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847259 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:40:14,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:40:14,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165686 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:40:17,758::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:40:17,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134019 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:40:24,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:40:24,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821808 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:40:27,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:40:27,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136003 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:40:34,721::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:40:34,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000840204 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:40:37,919::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:40:37,949::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828133 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:40:44,759::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:40:44,801::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150841 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:40:47,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:40:48,023::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000985894 s, 4.2 MB/s\n'; <rc> = 0
>Thread-15100::WARNING::2013-03-18 06:40:50,024::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:40:50,031::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 06:40:54,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:40:54,835::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000912708 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:40:58,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:40:58,109::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793676 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:41:00,035::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:41:00,036::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:41:00,535::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:41:00,541::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:41:00,543::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:41:04,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:41:04,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123256 s, 332 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:41:08,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:41:08,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000749721 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:41:14,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:41:14,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145369 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:41:18,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:41:18,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000778399 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:41:24,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:41:24,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153158 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:41:28,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:41:28,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109451 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:41:34,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:41:35,030::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168543 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:41:38,368::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:41:38,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00774249 s, 529 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:41:45,036::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:41:45,076::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156464 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:41:48,459::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:41:48,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102175 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:41:55,083::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:41:55,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00283398 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:41:58,530::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:41:58,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767093 s, 5.3 MB/s\n'; <rc> = 0
>Thread-15101::WARNING::2013-03-18 06:42:00,569::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:42:00,573::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 06:42:05,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:42:05,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833176 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:42:08,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:42:08,651::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147212 s, 2.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:42:10,575::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 06:42:10,576::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:42:10,578::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:42:11,030::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:42:11,037::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:42:11,038::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:42:15,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:42:15,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0160716 s, 255 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:42:18,691::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:42:18,739::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923447 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:42:25,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:42:25,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085222 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:42:28,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:42:28,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00108863 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:42:35,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:42:35,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013422 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:42:38,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:42:38,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142606 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:42:45,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:42:45,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807767 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:42:48,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:42:48,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126052 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:42:55,343::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:42:55,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136043 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:42:59,025::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 06:42:59,027::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 06:42:59,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:42:59,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00125929 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:42:59,070::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00125929 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 06:42:59,071::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 06:42:59,071::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 06:42:59,072::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 06:42:59,076::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:42:59,076::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:42:59,077::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:42:59,077::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:42:59,078::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:42:59,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:42:59,501::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:42:59,505::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:42:59,506::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 06:42:59,509::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 06:42:59,510::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 06:42:59,511::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 06:42:59,512::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:42:59,877::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:42:59,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:42:59,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119388 s, 343 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:43:05,384::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 06:43:05,385::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 06:43:05,386::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:43:05,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.015238 s, 134 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:43:05,432::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.015238 s, 134 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 06:43:05,432::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 06:43:05,433::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 06:43:05,435::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 06:43:05,436::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:43:05,437::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:43:05,438::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:43:05,438::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:43:05,439::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:43:05,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:43:05,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:43:05,873::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:43:05,874::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 06:43:05,876::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 06:43:05,877::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 06:43:05,878::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 06:43:05,879::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:43:06,278::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:43:06,280::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:43:06,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016665 s, 246 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:43:09,973::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:43:10,263::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0230559 s, 178 kB/s\n'; <rc> = 0
>Thread-15102::WARNING::2013-03-18 06:43:11,065::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:43:11,072::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 06:43:16,350::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:43:16,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0351222 s, 117 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:43:20,495::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:43:20,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160252 s, 2.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:43:21,076::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:43:21,077::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:43:21,571::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:43:21,577::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:43:21,579::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:43:26,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:43:26,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0147699 s, 277 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:43:30,577::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:43:30,602::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821422 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:43:36,465::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:43:36,507::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112585 s, 364 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:43:40,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:43:40,671::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000793409 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:43:46,514::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:43:46,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148276 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:43:50,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:43:50,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:43:56,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:43:56,605::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00983895 s, 416 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:44:00,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:44:00,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124843 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:44:06,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:44:06,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0410871 s, 99.7 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:44:10,885::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:44:10,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152508 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:44:16,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:44:16,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152071 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:44:21,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:44:21,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0232083 s, 176 kB/s\n'; <rc> = 0
>Thread-15103::WARNING::2013-03-18 06:44:21,607::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:44:21,613::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 06:44:26,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:44:26,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834927 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:44:31,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:44:31,666::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:44:31,684::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:44:31,710::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00403466 s, 1.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:44:32,186::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:44:32,191::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:44:32,192::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:44:36,812::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:44:36,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0260916 s, 157 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:44:41,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:44:41,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0249378 s, 164 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:44:46,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:44:46,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0138502 s, 296 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:44:51,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:44:51,944::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015218 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:44:56,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:44:56,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800031 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:45:01,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:45:02,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000800078 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:45:06,966::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:45:06,999::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139458 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:45:12,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:45:12,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00088901 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:45:17,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:45:17,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850726 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:45:22,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:45:22,173::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00169586 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:45:27,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:45:27,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139125 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:45:32,212::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-15104::WARNING::2013-03-18 06:45:32,241::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-20::DEBUG::2013-03-18 06:45:32,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163877 s, 2.5 MB/s\n'; <rc> = 0
>Thread-21::ERROR::2013-03-18 06:45:32,255::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 06:45:37,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:45:37,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814331 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:45:42,281::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:45:42,282::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:45:42,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:45:42,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0218843 s, 187 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:45:42,797::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:45:42,802::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:45:42,804::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:45:47,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:45:47,155::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102179 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:45:52,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:45:52,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000904297 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:45:57,161::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:45:57,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799765 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:46:02,558::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:46:02,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135461 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:46:07,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:46:07,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0189789 s, 216 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:46:12,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:46:12,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000727569 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:46:17,250::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:46:17,280::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000740015 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:46:22,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:46:22,753::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00171653 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:46:27,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:46:27,321::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000762499 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:46:32,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:46:32,828::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106008 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:46:37,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:46:37,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000922372 s, 4.4 MB/s\n'; <rc> = 0
>Thread-15105::WARNING::2013-03-18 06:46:42,833::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:46:42,839::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 06:46:42,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:46:42,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173888 s, 236 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:46:47,445::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:46:47,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00923225 s, 444 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:46:52,842::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:46:52,843::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:46:52,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:46:53,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130757 s, 313 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:46:53,353::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:46:53,360::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:46:53,362::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:46:57,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:46:57,541::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127143 s, 322 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:47:03,076::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:47:03,111::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116979 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:47:07,549::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:47:07,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146086 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:47:13,151::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:47:13,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795663 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:47:17,600::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:47:17,634::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000825839 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:47:23,220::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:47:23,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076463 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:47:27,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:47:27,672::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00183034 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:47:33,287::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:47:33,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836624 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:47:37,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:47:37,718::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141277 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:47:43,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:47:43,388::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837887 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:47:47,723::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:47:47,758::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920213 s, 4.5 MB/s\n'; <rc> = 0
>Thread-15106::WARNING::2013-03-18 06:47:53,394::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:47:53,398::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 06:47:53,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:47:53,457::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107198 s, 3.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:47:57,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:47:57,795::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000867136 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:48:03,400::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 06:48:03,401::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:48:03,402::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:48:03,497::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 06:48:03,500::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 06:48:03,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:48:03,560::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00805414 s, 254 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:48:03,562::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00805414 s, 254 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 06:48:03,562::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 06:48:03,563::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 06:48:03,565::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 06:48:03,566::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm reload operation' is holding the operation mutex, waiting...
>Thread-21::DEBUG::2013-03-18 06:48:03,849::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:48:03,855::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:48:03,855::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:48:03,893::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:48:03,896::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:48:03,900::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:48:03,900::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:48:03,901::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:48:03,902::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:48:04,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:48:04,305::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:48:04,306::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 06:48:04,309::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 06:48:04,310::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 06:48:04,311::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 06:48:04,312::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:48:04,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:48:04,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:48:04,723::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173246 s, 236 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:48:07,800::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 06:48:07,801::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 06:48:07,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:48:07,837::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00448143 s, 457 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:48:07,839::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00448143 s, 457 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 06:48:07,839::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 06:48:07,840::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 06:48:07,842::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 06:48:07,843::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:48:07,844::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:48:07,844::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:48:07,844::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:48:07,845::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:48:07,846::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:48:08,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:48:08,249::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:48:08,250::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 06:48:08,252::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 06:48:08,253::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 06:48:08,254::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 06:48:08,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:48:08,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:48:08,719::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:48:08,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133493 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:48:14,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:48:14,818::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123225 s, 332 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:48:18,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:48:18,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160276 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:48:24,857::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:48:24,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162299 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:48:28,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:48:28,853::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134635 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:48:34,943::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:48:34,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820298 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:48:38,859::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:48:38,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145686 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:48:45,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:48:45,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134461 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:48:48,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:48:48,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150362 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:48:55,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:48:55,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000754206 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:48:58,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:48:59,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184669 s, 222 kB/s\n'; <rc> = 0
>Thread-15107::WARNING::2013-03-18 06:49:03,900::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:49:03,906::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 06:49:05,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:49:05,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015466 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:49:09,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:49:09,054::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820194 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:49:13,908::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:49:13,909::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:49:14,344::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:49:14,352::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:49:14,354::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:49:15,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:49:15,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166403 s, 246 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:49:19,060::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:49:19,088::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797057 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:49:25,339::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:49:25,366::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816416 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:49:29,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:49:29,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147722 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:49:35,407::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:49:35,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0009163 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:49:39,143::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:49:39,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101461 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:49:45,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:49:45,512::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758948 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:49:49,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:49:49,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000773835 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:49:55,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:49:55,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814196 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:49:59,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:49:59,258::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162011 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:50:05,622::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:50:05,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146553 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:50:09,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:50:09,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129989 s, 3.2 MB/s\n'; <rc> = 0
>Thread-15108::WARNING::2013-03-18 06:50:14,400::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:50:14,407::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 06:50:15,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:50:15,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000748131 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:50:19,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:50:19,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872141 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:50:24,409::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:50:24,411::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:50:24,849::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:50:24,855::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:50:24,856::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:50:25,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:50:25,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0151143 s, 271 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:50:29,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:50:29,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897799 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:50:35,872::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:50:35,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151663 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:50:39,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:50:39,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152279 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:50:45,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:50:45,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812038 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:50:49,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:50:49,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850209 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:50:56,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:50:56,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822738 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:50:59,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:50:59,492::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000903603 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:51:06,099::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:51:06,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143956 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:51:09,497::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:51:09,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155912 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:51:16,181::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:51:16,222::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014114 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:51:19,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:51:19,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00096043 s, 4.3 MB/s\n'; <rc> = 0
>Thread-15109::WARNING::2013-03-18 06:51:24,885::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:51:24,891::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 06:51:26,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:51:26,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811532 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:51:29,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:51:29,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913997 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:51:34,894::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:51:34,895::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:51:35,299::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:51:35,305::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:51:35,307::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:51:36,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:51:36,382::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115807 s, 354 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:51:39,614::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:51:39,682::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000952267 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:51:46,422::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:51:46,472::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0211638 s, 194 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:51:49,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:51:49,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000826206 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:51:56,525::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:51:56,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00258133 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:51:59,732::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:51:59,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000824151 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:52:06,595::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:52:06,627::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000711704 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:52:09,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:52:09,809::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154416 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:52:16,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:52:16,692::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100182 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:52:19,814::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:52:19,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00163509 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:52:26,732::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:52:26,761::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858811 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:52:29,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:52:29,902::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138276 s, 3.0 MB/s\n'; <rc> = 0
>Thread-15110::WARNING::2013-03-18 06:52:35,332::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:52:35,337::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 06:52:36,802::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:52:36,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769594 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:52:39,909::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:52:39,955::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775804 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:52:45,339::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:52:45,340::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:52:45,767::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:52:45,774::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:52:45,775::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:52:46,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:52:46,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00479118 s, 855 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:52:49,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:52:49,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00293985 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:52:56,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:52:56,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000988835 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:52:59,995::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:53:00,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161421 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:53:07,029::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 06:53:07,031::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 06:53:07,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:53:07,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000819227 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:53:07,057::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000819227 s, 2.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 06:53:07,057::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 06:53:07,057::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 06:53:07,060::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 06:53:07,061::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:53:07,062::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:53:07,063::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:53:07,063::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:53:07,064::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:53:07,065::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:53:07,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:53:07,499::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:53:07,500::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 06:53:07,502::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 06:53:07,504::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 06:53:07,504::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 06:53:07,506::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:53:07,881::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:53:07,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:53:07,924::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169694 s, 241 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:53:10,039::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 06:53:10,041::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 06:53:10,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:53:10,079::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0133309 s, 154 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:53:10,081::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0133309 s, 154 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 06:53:10,082::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 06:53:10,082::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 06:53:10,084::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 06:53:10,085::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:53:10,086::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:53:10,086::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:53:10,087::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:53:10,087::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:53:10,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:53:10,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:53:10,492::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:53:10,493::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 06:53:10,496::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 06:53:10,497::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 06:53:10,497::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 06:53:10,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:53:10,946::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:53:10,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:53:10,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000859937 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:53:17,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:53:18,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122532 s, 334 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:53:20,989::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:53:21,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141006 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:53:28,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:53:28,094::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138685 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:53:31,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:53:31,069::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000804411 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:53:38,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:53:38,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00114247 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:53:41,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:53:41,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00117068 s, 3.5 MB/s\n'; <rc> = 0
>Thread-15111::WARNING::2013-03-18 06:53:45,801::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:53:45,807::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 06:53:48,217::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:53:48,244::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000926804 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:53:51,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:53:51,156::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008423 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:53:55,809::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 06:53:55,811::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:53:55,812::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:53:56,258::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:53:56,265::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:53:56,266::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:53:58,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:53:58,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175096 s, 234 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:54:01,162::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:54:01,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000934762 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:54:08,372::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:54:08,402::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872991 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:54:11,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:54:11,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080187 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:54:18,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:54:18,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146206 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:54:21,230::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:54:21,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788939 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:54:28,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:54:28,580::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807948 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:54:31,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:54:31,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149907 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:54:38,619::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:54:38,667::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177484 s, 231 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:54:41,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:54:41,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00306345 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:54:48,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:54:48,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168346 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:54:51,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:54:51,400::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0172514 s, 237 kB/s\n'; <rc> = 0
>Thread-15112::WARNING::2013-03-18 06:54:56,294::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:54:56,296::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 06:54:58,781::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:54:58,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821844 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:55:01,409::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:55:01,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770848 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:55:06,299::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:55:06,300::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:55:06,738::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:55:06,745::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:55:06,746::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:55:08,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:55:08,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173422 s, 236 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:55:11,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:55:11,494::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000913704 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:55:18,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:55:18,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834715 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:55:21,504::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:55:21,573::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0026153 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:55:29,002::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:55:29,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000914989 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:55:31,578::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:55:31,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.004253 s, 963 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:55:39,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:55:39,093::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833991 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:55:41,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:55:41,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000953591 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:55:49,133::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:55:49,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132468 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:55:51,741::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:55:51,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806907 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:55:59,222::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:55:59,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085837 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:56:01,805::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:56:01,881::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000703987 s, 5.8 MB/s\n'; <rc> = 0
>Thread-15113::WARNING::2013-03-18 06:56:06,770::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:56:06,774::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 06:56:09,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:56:09,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783697 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:56:11,889::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:56:11,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802464 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:56:16,779::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:56:16,780::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:56:17,199::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:56:17,205::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:56:17,207::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:56:19,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:56:19,411::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139582 s, 293 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:56:21,944::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:56:22,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791406 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:56:29,454::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:56:29,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839996 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:56:32,023::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:56:32,081::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00222137 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:56:39,530::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:56:39,572::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135259 s, 303 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:56:42,088::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:56:42,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0245603 s, 167 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:56:49,614::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:56:49,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00647197 s, 633 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:56:52,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:56:52,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000776157 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:56:59,693::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:56:59,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0009154 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:57:02,225::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:57:02,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147666 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:57:09,767::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:57:09,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179109 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:57:12,272::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:57:12,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154828 s, 2.6 MB/s\n'; <rc> = 0
>Thread-15114::WARNING::2013-03-18 06:57:17,237::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:57:17,241::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 06:57:19,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:57:19,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000797204 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:57:22,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:57:22,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000841269 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:57:27,243::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:57:27,244::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:57:27,657::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:57:27,663::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:57:27,665::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:57:29,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:57:29,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015173 s, 270 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:57:32,354::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:57:32,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127832 s, 320 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:57:40,012::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:57:40,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760247 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:57:42,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:57:42,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858317 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:57:50,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:57:50,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000733353 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:57:52,434::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:57:52,464::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000718525 s, 5.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:58:00,154::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:58:00,198::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159083 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:58:02,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:58:02,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142454 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:58:10,237::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 06:58:10,239::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 06:58:10,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:58:10,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000947023 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:58:10,268::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000947023 s, 2.2 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 06:58:10,269::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 06:58:10,269::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 06:58:10,271::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 06:58:10,272::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:58:10,272::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:58:10,273::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:58:10,273::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:58:10,274::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 06:58:10,275::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:58:10,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:58:10,710::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:58:10,711::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 06:58:10,714::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 06:58:10,716::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 06:58:10,716::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 06:58:10,717::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:58:11,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:58:11,081::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:58:11,139::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0241527 s, 170 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:58:12,522::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 06:58:12,523::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 06:58:12,524::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:58:12,577::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0173281 s, 118 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:58:12,578::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0173281 s, 118 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 06:58:12,579::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 06:58:12,579::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 06:58:12,580::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 06:58:12,581::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:58:12,582::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:58:12,583::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:58:12,586::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:58:12,586::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 06:58:12,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:58:13,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:58:13,065::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 06:58:13,066::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 06:58:13,068::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 06:58:13,069::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 06:58:13,070::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 06:58:13,071::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:58:13,466::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:58:13,468::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:58:13,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0136872 s, 299 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:58:21,180::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:58:21,261::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0395336 s, 104 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:58:23,527::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:58:23,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816052 s, 5.0 MB/s\n'; <rc> = 0
>Thread-15115::WARNING::2013-03-18 06:58:27,690::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:58:27,694::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 06:58:31,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:58:31,332::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865527 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:58:33,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:58:33,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145361 s, 2.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:58:37,696::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:58:37,697::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:58:38,127::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:58:38,133::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:58:38,135::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:58:41,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:58:41,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103398 s, 396 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:58:43,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:58:43,739::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0434288 s, 94.3 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:58:51,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:58:51,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164207 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:58:53,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:58:53,789::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00382273 s, 1.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:59:01,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:59:01,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084228 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:59:03,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:59:03,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835405 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:59:11,614::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:59:11,658::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153711 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:59:13,829::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:59:13,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805206 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:59:21,698::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:59:21,726::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818423 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:59:23,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:59:23,896::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000981887 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 06:59:31,765::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:59:31,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162685 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:59:33,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:59:33,934::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792326 s, 5.2 MB/s\n'; <rc> = 0
>Thread-15116::WARNING::2013-03-18 06:59:38,175::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 06:59:38,180::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 06:59:41,852::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:59:41,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000820558 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:59:43,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:59:43,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855996 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 06:59:48,182::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 06:59:48,183::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 06:59:48,184::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 06:59:48,644::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 06:59:48,652::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 06:59:48,653::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 06:59:51,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 06:59:51,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0141957 s, 289 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 06:59:53,972::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 06:59:54,010::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00743996 s, 551 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:00:02,024::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:00:02,081::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0265925 s, 154 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:00:04,015::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:00:04,058::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140451 s, 292 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:00:12,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:00:12,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129969 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:00:14,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:00:14,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785182 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:00:22,205::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:00:22,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000835239 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:00:24,098::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:00:24,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118491 s, 346 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:00:32,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:00:32,308::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000770981 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:00:34,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:00:34,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000980292 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:00:42,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:00:42,387::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154718 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:00:44,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:00:44,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000944285 s, 4.3 MB/s\n'; <rc> = 0
>Thread-15117::WARNING::2013-03-18 07:00:48,684::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:00:48,689::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:00:52,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:00:52,455::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837917 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:00:54,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:00:54,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0064279 s, 637 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:00:58,694::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:00:58,695::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:00:59,155::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:00:59,161::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:00:59,163::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:01:02,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:01:02,602::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0511487 s, 80.1 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:01:04,280::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:01:04,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102987 s, 398 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:01:12,642::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:01:12,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145093 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:01:14,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:01:14,350::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000807228 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:01:22,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:01:22,754::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788868 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:01:24,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:01:24,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100309 s, 4.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:01:32,796::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:01:32,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131144 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:01:34,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:01:34,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000920781 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:01:42,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:01:42,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146328 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:01:44,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:01:44,655::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0229278 s, 179 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:01:52,948::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:01:52,979::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812314 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:01:54,679::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:01:54,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137183 s, 299 kB/s\n'; <rc> = 0
>Thread-15118::WARNING::2013-03-18 07:01:59,189::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:01:59,192::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:02:03,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:02:03,049::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814639 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:02:04,738::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:02:04,777::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104394 s, 392 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:02:09,201::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:02:09,202::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:02:09,623::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:02:09,630::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:02:09,631::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:02:13,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:02:13,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00774201 s, 529 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:02:14,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:02:14,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0181418 s, 226 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:02:23,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:02:23,195::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881558 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:02:24,837::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:02:24,901::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0217629 s, 188 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:02:33,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:02:33,270::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00073591 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:02:34,914::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:02:34,963::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000965882 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:02:43,310::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:02:43,340::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085783 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:02:44,969::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:02:45,018::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156599 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:02:53,379::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:02:53,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145956 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:02:55,028::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:02:55,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160343 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:03:03,470::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:03:03,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161951 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:03:05,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:03:05,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000870879 s, 4.7 MB/s\n'; <rc> = 0
>Thread-15119::WARNING::2013-03-18 07:03:09,661::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:03:09,667::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:03:13,565::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 07:03:13,567::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 07:03:13,567::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:03:13,609::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00155709 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:03:13,610::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00155709 s, 1.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 07:03:13,611::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 07:03:13,611::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 07:03:13,613::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 07:03:13,614::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:03:13,615::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:03:13,615::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:03:13,615::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:03:13,616::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:03:13,617::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:03:14,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:03:14,035::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:03:14,036::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 07:03:14,038::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 07:03:14,039::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 07:03:14,039::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 07:03:14,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:03:14,391::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:03:14,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:03:14,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0196672 s, 208 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:03:15,142::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 07:03:15,144::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 07:03:15,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:03:15,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00110137 s, 1.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:03:15,174::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00110137 s, 1.9 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 07:03:15,174::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 07:03:15,174::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 07:03:15,177::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 07:03:15,178::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:03:15,179::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:03:15,179::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:03:15,180::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:03:15,180::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:03:15,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:03:15,569::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:03:15,574::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:03:15,575::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 07:03:15,577::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 07:03:15,578::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 07:03:15,579::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 07:03:15,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:03:15,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:03:15,940::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:03:15,979::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104135 s, 393 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:03:19,670::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:03:19,671::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:03:20,122::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:03:20,128::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:03:20,129::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:03:24,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:03:24,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0242188 s, 169 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:03:25,985::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:03:26,015::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000967944 s, 4.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:03:34,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:03:34,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795307 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:03:36,021::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:03:36,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00184742 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:03:44,646::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:03:44,677::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000717692 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:03:46,057::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:03:46,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000860732 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:03:54,717::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:03:54,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131228 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:03:56,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:03:56,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00286576 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:04:04,799::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:04:04,842::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0137135 s, 299 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:04:06,129::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:04:06,168::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00184555 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:04:14,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:04:14,922::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079573 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:04:16,173::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:04:16,213::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145193 s, 2.8 MB/s\n'; <rc> = 0
>Thread-15120::WARNING::2013-03-18 07:04:20,156::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:04:20,159::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:04:24,968::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:04:24,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936754 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:04:26,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:04:26,249::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000775896 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:04:30,163::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:04:30,164::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:04:30,689::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:04:30,695::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:04:30,697::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:04:35,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:04:35,083::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0131164 s, 312 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:04:36,254::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:04:36,299::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159034 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:04:45,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:04:45,151::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000953497 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:04:46,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:04:46,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00212747 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:04:55,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:04:55,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000928039 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:04:56,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:04:56,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000811993 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:05:05,259::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:05:05,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000970949 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:05:06,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:05:06,399::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141469 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:05:15,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:05:15,363::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774253 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:05:16,404::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:05:16,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162063 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:05:25,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:05:25,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00177779 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:05:26,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:05:26,489::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145899 s, 2.8 MB/s\n'; <rc> = 0
>Thread-15121::WARNING::2013-03-18 07:05:30,722::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:05:30,725::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:05:35,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:05:35,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00095321 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:05:36,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:05:36,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829021 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:05:40,730::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 07:05:40,731::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:05:40,732::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:05:41,156::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:05:41,162::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:05:41,164::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:05:45,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:05:45,617::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146301 s, 280 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:05:46,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:05:46,555::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743001 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:05:55,657::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:05:55,690::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014788 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:05:56,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:05:56,602::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151011 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:06:05,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:06:05,756::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080415 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:06:06,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:06:06,637::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101664 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:06:15,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:06:15,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848135 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:06:16,644::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:06:16,684::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00423889 s, 966 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:06:25,866::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:06:25,895::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854717 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:06:26,689::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:06:26,746::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129348 s, 317 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:06:35,938::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:06:35,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160132 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:06:36,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:06:36,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152675 s, 2.7 MB/s\n'; <rc> = 0
>Thread-15122::WARNING::2013-03-18 07:06:41,190::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:06:41,194::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:06:46,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:06:46,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000921744 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:06:46,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:06:46,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000657323 s, 6.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:06:51,196::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:06:51,197::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:06:51,616::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:06:51,622::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:06:51,624::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:06:56,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:06:56,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0173522 s, 236 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:06:56,882::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:06:56,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0341631 s, 120 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:07:06,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:07:06,216::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864919 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:07:06,963::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:07:07,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0039585 s, 1.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:07:16,258::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:07:16,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794329 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:07:17,031::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:07:17,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174751 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:07:26,331::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:07:26,375::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00264164 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:07:27,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:07:27,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00212749 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:07:36,417::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:07:36,449::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013586 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:07:37,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:07:37,177::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140157 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:07:46,499::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:07:46,551::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00124025 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:07:47,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:07:47,238::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00287015 s, 1.4 MB/s\n'; <rc> = 0
>Thread-15123::WARNING::2013-03-18 07:07:51,656::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:07:51,660::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:07:56,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:07:56,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000914938 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:07:57,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:07:57,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107387 s, 3.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:08:01,662::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:08:01,663::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:08:02,160::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:08:02,167::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:08:02,168::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:08:06,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:08:06,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0548612 s, 74.7 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:08:07,279::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:08:07,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166981 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:08:16,805::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 07:08:16,806::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 07:08:16,807::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:08:16,854::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0014209 s, 1.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:08:16,857::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0014209 s, 1.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 07:08:16,862::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 07:08:16,864::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 07:08:16,866::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 07:08:16,867::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:08:16,868::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:08:16,869::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:08:16,869::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:08:16,869::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:08:16,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:08:17,253::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:08:17,258::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:08:17,259::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 07:08:17,261::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 07:08:17,263::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 07:08:17,263::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 07:08:17,265::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:08:17,351::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 07:08:17,354::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 07:08:17,355::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:08:17,417::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00129496 s, 1.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:08:17,419::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00129496 s, 1.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 07:08:17,419::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 07:08:17,420::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 07:08:17,421::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 07:08:17,422::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:08:17,423::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:08:17,424::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:08:17,424::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:08:17,424::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:08:17,425::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:08:17,734::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:08:17,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:08:17,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0139259 s, 294 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:08:17,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:08:17,867::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:08:17,868::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 07:08:17,870::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 07:08:17,871::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 07:08:17,872::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 07:08:17,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:08:18,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:08:18,254::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:08:18,297::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0112524 s, 364 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:08:27,845::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:08:27,875::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802138 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:08:28,304::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:08:28,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147717 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:08:37,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:08:37,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810807 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:08:38,359::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:08:38,397::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00242265 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:08:47,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:08:48,027::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148147 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:08:48,403::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:08:48,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00311576 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:08:58,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:08:58,101::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111235 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:08:58,457::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:08:58,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0016123 s, 2.5 MB/s\n'; <rc> = 0
>Thread-15124::WARNING::2013-03-18 07:09:02,197::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:09:02,200::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:09:08,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:09:08,174::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847836 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:09:08,516::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:09:08,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000942017 s, 4.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:09:12,202::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:09:12,203::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:09:12,625::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:09:12,632::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:09:12,633::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:09:18,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:09:18,260::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0178926 s, 229 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:09:18,552::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:09:18,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000834304 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:09:28,300::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:09:28,325::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791382 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:09:28,584::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:09:28,628::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014789 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:09:38,372::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:09:38,399::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008354 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:09:38,633::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:09:38,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168865 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:09:48,440::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:09:48,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000659269 s, 6.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:09:48,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:09:48,716::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148127 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:09:58,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:09:58,540::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00281499 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:09:58,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:09:58,766::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156337 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:10:08,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:10:08,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139769 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:10:08,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:10:08,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00221627 s, 1.8 MB/s\n'; <rc> = 0
>Thread-15125::WARNING::2013-03-18 07:10:12,664::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:10:12,669::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:10:18,666::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:10:18,692::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000760867 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:10:18,824::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:10:18,852::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847182 s, 4.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:10:22,671::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:10:22,673::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:10:23,088::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:10:23,095::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:10:23,096::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:10:28,737::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:10:28,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0169794 s, 241 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:10:28,858::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:10:28,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00217396 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:10:38,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:10:38,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174728 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:10:38,907::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:10:38,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118032 s, 3.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:10:48,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:10:49,019::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:10:49,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.013998 s, 293 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:10:49,054::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109527 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:10:59,072::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:10:59,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:10:59,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129114 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:10:59,190::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00105317 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:11:09,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:11:09,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:11:09,259::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828699 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:11:09,289::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145736 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:11:19,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:11:19,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:11:19,404::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015672 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:11:19,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000736634 s, 5.6 MB/s\n'; <rc> = 0
>Thread-15126::WARNING::2013-03-18 07:11:23,120::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:11:23,126::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 07:11:29,426::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:11:29,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:11:29,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00186068 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:11:29,497::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000903461 s, 4.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:11:33,129::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 07:11:33,130::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:11:33,130::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:11:33,575::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:11:33,583::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:11:33,585::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:11:39,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:11:39,535::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878196 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:11:39,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:11:39,594::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0219487 s, 187 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:11:49,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:11:49,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000671546 s, 6.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:11:49,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:11:49,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0104458 s, 392 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:11:59,625::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:11:59,671::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135211 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:11:59,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:11:59,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173392 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:12:09,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:12:09,720::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154741 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:12:09,791::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:12:09,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869286 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:12:19,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:12:19,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145463 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:12:19,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:12:19,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135834 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:12:29,775::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:12:29,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154236 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:12:29,947::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:12:29,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153327 s, 2.7 MB/s\n'; <rc> = 0
>Thread-15127::WARNING::2013-03-18 07:12:33,619::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:12:33,626::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 07:12:39,828::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:12:39,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897242 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:12:40,038::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:12:40,081::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174569 s, 2.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:12:43,634::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:12:43,635::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:12:44,075::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:12:44,083::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:12:44,085::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:12:49,870::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:12:49,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140388 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:12:50,121::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:12:50,166::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00854524 s, 479 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:12:59,906::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:12:59,936::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00229189 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:13:00,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:13:00,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015939 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:13:09,941::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:13:09,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000769294 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:13:10,288::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:13:10,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133694 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:13:19,978::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 07:13:19,980::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 07:13:19,980::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:13:20,035::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000803063 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:13:20,036::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000803063 s, 2.6 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 07:13:20,037::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 07:13:20,037::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 07:13:20,039::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 07:13:20,040::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:13:20,040::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:13:20,041::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:13:20,041::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:13:20,041::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:13:20,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:13:20,363::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 07:13:20,366::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 07:13:20,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:13:20,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0504842 s, 40.6 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:13:20,448::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0504842 s, 40.6 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 07:13:20,448::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 07:13:20,448::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 07:13:20,450::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 07:13:20,452::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm reload operation' is holding the operation mutex, waiting...
>Thread-22::DEBUG::2013-03-18 07:13:20,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:13:20,553::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:13:20,557::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-20::DEBUG::2013-03-18 07:13:20,558::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:13:20,562::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:13:20,571::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::WARNING::2013-03-18 07:13:20,570::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-20::DEBUG::2013-03-18 07:13:20,572::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::WARNING::2013-03-18 07:13:20,572::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-20::DEBUG::2013-03-18 07:13:20,574::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::WARNING::2013-03-18 07:13:20,574::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 07:13:20,581::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:13:20,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:13:20,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:13:20,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:13:21,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0130613 s, 314 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:13:21,071::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:13:21,074::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:13:21,075::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 07:13:21,081::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 07:13:21,083::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 07:13:21,083::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 07:13:21,084::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:13:21,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:13:21,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:13:21,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00629046 s, 651 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:13:31,025::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:13:31,062::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874959 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:13:31,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:13:31,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139506 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:13:41,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:13:41,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151112 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:13:41,626::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:13:41,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147719 s, 2.8 MB/s\n'; <rc> = 0
>Thread-15128::WARNING::2013-03-18 07:13:44,115::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:13:44,119::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 07:13:51,119::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:13:51,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137229 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:13:51,709::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:13:51,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162746 s, 2.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:13:54,121::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:13:54,123::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:13:54,518::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:13:54,524::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:13:54,526::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:14:01,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:14:01,208::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00183303 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:14:01,795::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:14:01,847::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132511 s, 309 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:14:11,215::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:14:11,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00166436 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:14:12,052::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:14:12,080::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844086 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:14:21,252::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:14:21,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000863364 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:14:22,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:14:22,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809149 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:14:31,286::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:14:31,317::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00109203 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:14:32,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:14:32,215::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0022686 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:14:41,324::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:14:41,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141694 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:14:42,261::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:14:42,299::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828663 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:14:51,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:14:51,403::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137021 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:14:52,340::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:14:52,381::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147906 s, 2.8 MB/s\n'; <rc> = 0
>Thread-15129::WARNING::2013-03-18 07:14:54,553::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:14:54,558::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 07:15:01,414::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:15:01,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000954775 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:15:02,429::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:15:02,460::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833396 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:15:04,571::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:15:04,572::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:15:05,016::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:15:05,023::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:15:05,024::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:15:11,446::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:15:11,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900979 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:15:12,501::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:15:12,548::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166519 s, 246 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:15:21,482::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:15:21,512::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107451 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:15:22,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:15:22,613::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000891554 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:15:31,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:15:31,564::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170712 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:15:32,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:15:32,695::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157565 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:15:41,569::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:15:41,611::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134829 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:15:42,734::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:15:42,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146239 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:15:51,617::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:15:51,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147958 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:15:52,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:15:52,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136261 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:16:01,661::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:16:01,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128799 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:16:02,900::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:16:02,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00067113 s, 6.1 MB/s\n'; <rc> = 0
>Thread-15130::WARNING::2013-03-18 07:16:05,049::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:16:05,053::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 07:16:11,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:16:11,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783271 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:16:12,990::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:16:13,024::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000890171 s, 4.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:16:15,056::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:16:15,057::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:16:15,509::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:16:15,515::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:16:15,517::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:16:21,742::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:16:21,768::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000789138 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:16:23,063::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:16:23,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00928409 s, 441 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:16:31,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:16:31,816::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160358 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:16:33,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:16:33,170::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000908194 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:16:41,821::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:16:41,851::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777421 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:16:43,213::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:16:43,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014577 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:16:51,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:16:51,898::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156626 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:16:53,293::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:16:53,357::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0195405 s, 210 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:17:01,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:17:01,933::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752959 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:17:03,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:17:03,429::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000772985 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:17:11,938::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:17:11,981::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013841 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:17:13,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:17:13,512::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138222 s, 3.0 MB/s\n'; <rc> = 0
>Thread-15131::WARNING::2013-03-18 07:17:15,542::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:17:15,546::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 07:17:21,987::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:17:22,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000817263 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:17:23,554::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:17:23,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0425637 s, 96.2 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:17:25,547::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 07:17:25,548::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:17:25,549::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:17:25,994::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:17:26,000::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:17:26,002::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:17:32,021::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:17:32,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000903022 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:17:33,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:17:33,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134985 s, 303 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:17:42,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:17:42,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909138 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:17:43,742::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:17:43,769::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128999 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:17:52,092::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:17:52,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845174 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:17:53,809::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:17:53,853::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150661 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:18:02,130::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:18:02,172::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138273 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:18:03,892::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:18:03,938::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000781807 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:18:12,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:18:12,210::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794813 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:18:13,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:18:14,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00339076 s, 1.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:18:22,214::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 07:18:22,222::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 07:18:22,224::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:18:22,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000870132 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:18:22,258::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000870132 s, 2.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 07:18:22,259::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 07:18:22,259::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 07:18:22,261::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 07:18:22,263::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:18:22,264::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:18:22,264::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:18:22,264::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:18:22,265::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:18:22,266::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:18:22,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:18:22,722::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:18:22,723::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 07:18:22,727::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 07:18:22,728::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 07:18:22,729::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 07:18:22,730::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:18:23,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:18:23,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:18:23,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143961 s, 285 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:18:24,051::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 07:18:24,054::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 07:18:24,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:18:24,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0209814 s, 97.6 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:18:24,139::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0209814 s, 97.6 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 07:18:24,139::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 07:18:24,144::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 07:18:24,146::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 07:18:24,150::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:18:24,151::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:18:24,152::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:18:24,154::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:18:24,154::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:18:24,156::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:18:24,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:18:24,681::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:18:24,682::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 07:18:24,685::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 07:18:24,687::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 07:18:24,687::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 07:18:24,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:18:25,550::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:18:25,563::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:18:25,619::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00735041 s, 557 kB/s\n'; <rc> = 0
>Thread-15132::WARNING::2013-03-18 07:18:26,026::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:18:26,029::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 07:18:33,148::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:18:33,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0152393 s, 269 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:18:35,843::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:18:35,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000751639 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:18:36,035::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:18:36,036::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:18:36,494::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:18:36,501::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:18:36,502::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:18:43,199::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:18:43,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838502 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:18:45,913::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:18:45,954::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836102 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:18:53,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:18:53,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794663 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:18:56,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:18:56,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00395293 s, 1.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:19:03,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:19:03,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00176806 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:19:06,125::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:19:06,158::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000712117 s, 5.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:19:13,319::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:19:13,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000742742 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:19:16,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:19:16,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0144621 s, 283 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:19:23,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:19:23,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138441 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:19:26,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:19:26,330::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00093969 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:19:33,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:19:33,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818898 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:19:36,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:19:36,413::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0110745 s, 370 kB/s\n'; <rc> = 0
>Thread-15133::WARNING::2013-03-18 07:19:36,530::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:19:36,534::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 07:19:43,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:19:43,481::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079925 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:19:46,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:19:46,575::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:19:46,578::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:19:46,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.033852 s, 121 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:19:47,091::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:19:47,096::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:19:47,097::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:19:53,492::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:19:53,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00151297 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:19:56,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:19:56,818::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0332059 s, 123 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:20:03,543::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:20:03,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139336 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:20:06,859::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:20:06,885::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909076 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:20:13,579::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:20:13,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00081756 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:20:16,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:20:16,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00275837 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:20:23,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:20:23,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141039 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:20:27,014::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:20:27,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136545 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:20:33,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:20:33,701::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086429 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:20:37,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:20:37,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136072 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:20:43,706::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:20:43,740::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153034 s, 2.7 MB/s\n'; <rc> = 0
>Thread-15134::WARNING::2013-03-18 07:20:47,123::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:20:47,127::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:20:47,169::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:20:47,194::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000909319 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:20:53,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:20:53,777::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000959903 s, 4.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:20:57,129::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:20:57,130::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:20:57,237::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:20:57,301::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163417 s, 251 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:20:57,617::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:20:57,622::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:20:57,623::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:21:03,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:21:03,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175768 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:21:07,369::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:21:07,406::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115806 s, 354 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:21:13,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:21:13,860::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091743 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:21:17,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:21:17,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158927 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:21:23,865::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:21:23,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152216 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:21:27,531::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:21:27,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00277336 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:21:33,910::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:21:33,939::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00188096 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:21:37,630::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:21:37,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0333203 s, 123 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:21:43,944::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:21:43,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142032 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:21:47,779::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:21:47,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848619 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:21:53,988::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:21:54,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140521 s, 2.9 MB/s\n'; <rc> = 0
>Thread-15135::WARNING::2013-03-18 07:21:57,646::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:21:57,650::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:21:57,894::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:21:57,956::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00174458 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:22:04,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:22:04,082::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0103008 s, 398 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:22:07,653::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:22:07,654::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:22:07,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:22:08,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148038 s, 2.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:22:08,116::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:22:08,122::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:22:08,123::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:22:14,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:22:14,132::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137823 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:22:18,106::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:22:18,133::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795392 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:22:24,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:22:24,184::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0182165 s, 225 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:22:28,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:22:28,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000785502 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:22:34,190::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:22:34,233::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126143 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:22:38,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:22:38,284::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014642 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:22:44,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:22:44,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013794 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:22:48,328::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:22:48,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143632 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:22:54,290::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:22:54,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000810186 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:22:58,412::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:22:58,440::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000756748 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:23:04,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:23:04,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753864 s, 5.4 MB/s\n'; <rc> = 0
>Thread-15136::WARNING::2013-03-18 07:23:08,153::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:23:08,155::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:23:08,482::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:23:08,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000786578 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:23:14,374::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:23:14,401::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000771685 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:23:18,157::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 07:23:18,158::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:23:18,159::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:23:18,561::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:23:18,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158994 s, 2.6 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:23:18,657::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:23:18,662::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:23:18,666::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:23:24,407::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 07:23:24,410::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 07:23:24,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:23:24,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000736191 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:23:24,438::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000736191 s, 2.8 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 07:23:24,438::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 07:23:24,438::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 07:23:24,440::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 07:23:24,441::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:23:24,442::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:23:24,442::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:23:24,442::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:23:24,443::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:23:24,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:23:24,861::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:23:24,869::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:23:24,870::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 07:23:24,872::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 07:23:24,874::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 07:23:24,874::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 07:23:24,876::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:23:25,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:23:25,239::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:23:25,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0025837 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:23:28,688::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 07:23:28,689::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 07:23:28,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:23:28,717::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000863728 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:23:28,719::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000863728 s, 2.4 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 07:23:28,720::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 07:23:28,720::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 07:23:28,723::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 07:23:28,724::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:23:28,724::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:23:28,725::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:23:28,725::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:23:28,726::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:23:28,727::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:23:29,149::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:23:29,153::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:23:29,154::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 07:23:29,157::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 07:23:29,158::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 07:23:29,158::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 07:23:29,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:23:29,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:23:29,534::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:23:29,591::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0282039 s, 145 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:23:35,276::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:23:35,339::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00694309 s, 590 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:23:39,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:23:39,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146924 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:23:45,344::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:23:45,386::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000758292 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:23:49,725::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:23:49,766::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138028 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:23:55,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:23:55,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000745785 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:23:59,806::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:23:59,849::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139678 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:24:05,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:24:05,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119348 s, 3.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:24:09,890::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:24:09,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000736195 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:24:15,472::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:24:15,512::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122968 s, 3.3 MB/s\n'; <rc> = 0
>Thread-15137::WARNING::2013-03-18 07:24:18,710::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:24:18,716::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:24:19,960::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:24:19,985::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102526 s, 4.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:24:25,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:24:25,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142955 s, 2.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:24:28,720::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:24:28,721::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:24:29,112::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:24:29,119::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:24:29,120::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:24:30,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:24:30,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0236115 s, 173 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:24:35,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:24:35,604::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000882505 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:24:40,117::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:24:40,146::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00118942 s, 3.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:24:45,610::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:24:45,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135245 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:24:50,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:24:50,220::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153804 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:24:55,672::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:24:55,712::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842688 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:25:00,260::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:25:00,287::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000720614 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:25:05,717::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:25:05,766::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000801915 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:25:10,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:25:10,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138361 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:25:15,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:25:15,812::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000910252 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:25:20,411::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:25:20,475::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0239587 s, 171 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:25:25,817::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:25:25,873::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145527 s, 2.8 MB/s\n'; <rc> = 0
>Thread-15138::WARNING::2013-03-18 07:25:29,148::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:25:29,150::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:25:30,515::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:25:30,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086495 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:25:35,880::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:25:35,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000832507 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:25:39,152::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:25:39,154::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:25:39,578::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:25:39,584::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:25:39,586::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:25:40,587::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:25:40,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000831159 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:25:45,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:25:45,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0025605 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:25:50,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:25:50,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798976 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:25:55,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:25:55,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000816239 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:26:00,728::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:26:00,774::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00158805 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:26:06,005::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:26:06,061::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00106588 s, 3.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:26:10,813::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:26:10,855::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140607 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:26:16,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:26:16,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00122443 s, 3.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:26:20,894::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:26:20,937::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160033 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:26:26,120::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:26:26,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847667 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:26:30,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:26:31,006::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000944579 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:26:36,157::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:26:36,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136995 s, 3.0 MB/s\n'; <rc> = 0
>Thread-15139::WARNING::2013-03-18 07:26:39,613::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:26:39,620::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:26:41,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:26:41,075::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833849 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:26:46,195::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:26:46,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000961164 s, 4.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:26:49,623::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:26:49,624::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:26:50,053::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:26:50,060::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:26:50,062::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:26:51,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:26:51,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000739794 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:26:56,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:26:56,293::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0341302 s, 120 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:27:01,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:27:01,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132681 s, 3.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:27:06,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:27:06,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134934 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:27:11,267::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:27:11,294::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777408 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:27:16,353::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:27:16,385::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000961027 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:27:21,335::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:27:21,360::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837804 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:27:26,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:27:26,433::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134345 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:27:31,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:27:31,444::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015815 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:27:36,444::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:27:36,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143464 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:27:41,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:27:41,528::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135806 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:27:46,488::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:27:46,537::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00816688 s, 502 kB/s\n'; <rc> = 0
>Thread-15140::WARNING::2013-03-18 07:27:50,092::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:27:50,100::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:27:51,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:27:51,624::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0272316 s, 150 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:27:56,547::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:27:56,576::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000874942 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:28:00,112::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:28:00,113::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:28:00,546::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:28:00,553::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:28:00,554::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:28:01,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:28:01,711::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.016432 s, 249 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:28:06,582::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:28:06,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013519 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:28:11,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:28:11,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000784458 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:28:16,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:28:16,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144249 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:28:21,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:28:21,876::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145437 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:28:26,658::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 07:28:26,659::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 07:28:26,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:28:26,687::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00089911 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:28:26,698::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00089911 s, 2.3 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 07:28:26,698::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 07:28:26,698::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 07:28:26,700::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 07:28:26,701::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:28:26,701::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:28:26,702::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:28:26,702::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:28:26,703::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:28:26,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:28:27,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:28:27,116::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:28:27,117::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 07:28:27,119::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 07:28:27,121::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 07:28:27,121::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 07:28:27,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:28:27,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:28:27,476::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:28:27,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899036 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:28:31,914::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 07:28:31,916::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 07:28:31,916::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:28:31,962::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00136603 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:28:31,963::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00136603 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 07:28:31,964::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 07:28:31,964::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 07:28:31,965::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 07:28:31,967::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:28:31,969::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:28:31,970::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:28:31,970::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:28:31,970::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:28:31,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:28:32,396::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:28:32,400::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:28:32,401::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 07:28:32,403::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 07:28:32,405::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 07:28:32,405::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 07:28:32,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:28:32,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:28:32,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:28:32,805::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00636099 s, 644 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:28:37,511::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:28:37,546::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00332794 s, 1.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:28:42,849::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:28:42,914::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0242407 s, 169 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:28:47,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:28:47,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087975 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:28:52,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:28:52,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159938 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:28:57,586::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:28:57,620::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00116665 s, 3.5 MB/s\n'; <rc> = 0
>Thread-15141::WARNING::2013-03-18 07:29:00,580::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:29:00,584::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:29:03,035::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:29:03,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916787 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:29:07,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:29:07,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000934096 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:29:10,593::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 07:29:10,594::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:29:10,595::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:29:11,094::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:29:11,100::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:29:11,102::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:29:13,107::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:29:13,150::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153241 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:29:17,662::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:29:17,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133636 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:29:23,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:29:23,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000919661 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:29:27,711::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:29:27,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142732 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:29:33,255::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:29:33,287::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000783138 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:29:37,751::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:29:37,779::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000812118 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:29:43,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:29:43,358::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000759193 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:29:47,785::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:29:47,814::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000899639 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:29:53,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:29:53,439::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160502 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:29:57,819::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:29:57,850::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000828174 s, 4.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:30:03,478::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:30:03,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000823518 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:30:07,855::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:30:07,887::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000802596 s, 5.1 MB/s\n'; <rc> = 0
>Thread-15142::WARNING::2013-03-18 07:30:11,125::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:30:11,129::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:30:13,556::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:30:13,597::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160311 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:30:17,897::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:30:17,926::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872301 s, 4.7 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:30:21,131::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:30:21,132::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:30:21,557::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:30:21,564::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:30:21,565::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:30:23,639::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:30:23,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000849359 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:30:27,932::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:30:27,979::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131504 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:30:33,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:30:33,750::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141136 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:30:37,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:30:38,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815258 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:30:43,789::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:30:43,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00074814 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:30:48,020::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:30:48,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787878 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:30:53,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:30:53,894::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000952258 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:30:58,056::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:30:58,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014292 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:31:03,933::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:31:03,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139529 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:31:08,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:31:08,147::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015537 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:31:14,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:31:14,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00123088 s, 3.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:31:18,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:31:18,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125691 s, 3.3 MB/s\n'; <rc> = 0
>Thread-15143::WARNING::2013-03-18 07:31:21,591::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:31:21,595::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:31:24,103::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:31:24,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000805241 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:31:28,198::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:31:28,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000936817 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:31:31,597::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:31:31,599::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:31:32,028::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:31:32,035::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:31:32,036::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:31:34,167::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:31:34,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000721758 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:31:38,234::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:31:38,277::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00173016 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:31:44,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:31:44,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0109334 s, 375 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:31:48,282::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:31:48,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149892 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:31:54,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:31:54,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897315 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:31:58,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:31:58,362::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143477 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:32:04,375::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:32:04,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00167751 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:32:08,367::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:32:08,410::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160515 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:32:14,463::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:32:14,503::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175115 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:32:18,420::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:32:18,451::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000875449 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:32:24,545::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:32:24,584::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162822 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:32:28,462::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:32:28,495::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0035681 s, 1.1 MB/s\n'; <rc> = 0
>Thread-15144::WARNING::2013-03-18 07:32:32,061::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:32:32,065::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:32:34,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:32:34,671::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153701 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:32:38,505::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:32:38,534::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000976523 s, 4.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:32:42,074::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:32:42,075::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:32:42,523::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:32:42,529::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:32:42,531::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:32:44,715::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:32:44,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00089853 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:32:48,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:32:48,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798299 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:32:54,787::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:32:54,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000933394 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:32:58,572::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:32:58,606::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000892111 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:33:04,854::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:33:04,883::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000883196 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:33:08,611::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:33:08,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129959 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:33:14,923::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:33:14,963::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159085 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:33:18,660::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:33:18,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00143419 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:33:25,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:33:25,047::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134769 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:33:28,708::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 07:33:28,710::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 07:33:28,710::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:33:28,742::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000768578 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:33:28,744::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000768578 s, 2.7 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 07:33:28,744::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 07:33:28,744::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 07:33:28,746::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 07:33:28,747::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:33:28,747::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:33:28,748::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:33:28,748::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:33:28,749::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:33:28,750::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:33:29,186::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:33:29,190::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:33:29,191::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 07:33:29,194::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 07:33:29,195::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 07:33:29,195::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 07:33:29,197::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:33:29,562::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:33:29,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:33:29,629::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0119123 s, 344 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:33:35,085::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 07:33:35,088::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 07:33:35,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:33:35,126::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00131403 s, 1.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:33:35,131::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00131403 s, 1.6 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 07:33:35,131::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 07:33:35,131::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 07:33:35,133::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 07:33:35,134::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:33:35,135::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:33:35,135::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:33:35,135::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:33:35,136::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:33:35,138::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:33:35,543::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:33:35,547::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:33:35,549::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 07:33:35,556::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 07:33:35,557::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 07:33:35,557::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 07:33:35,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:33:35,982::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:33:35,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:33:36,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00708526 s, 578 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:33:39,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:33:39,696::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761742 s, 5.4 MB/s\n'; <rc> = 0
>Thread-15145::WARNING::2013-03-18 07:33:42,571::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:33:42,577::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:33:46,077::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:33:46,106::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000822229 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:33:49,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:33:49,766::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00121795 s, 3.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:33:52,580::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:33:52,581::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:33:53,008::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:33:53,015::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:33:53,016::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:33:56,149::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:33:56,189::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00382683 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:33:59,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:33:59,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00555375 s, 738 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:34:06,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:34:06,274::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.015634 s, 262 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:34:09,827::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:34:09,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131492 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:34:16,314::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:34:16,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00164438 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:34:19,883::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:34:19,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000923091 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:34:26,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:34:26,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808728 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:34:29,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:34:29,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0008188 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:34:36,466::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:34:36,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134907 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:34:39,953::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:34:39,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00139581 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:34:46,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:34:46,593::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135161 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:34:49,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:34:50,042::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162229 s, 2.5 MB/s\n'; <rc> = 0
>Thread-15146::WARNING::2013-03-18 07:34:53,040::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:34:53,043::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:34:56,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:34:56,659::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837873 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:35:00,050::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:35:00,092::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774657 s, 5.3 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:35:03,088::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 07:35:03,089::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:35:03,090::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:35:03,525::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:35:03,532::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:35:03,533::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:35:06,703::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:35:06,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000884966 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:35:10,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:35:10,135::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000855316 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:35:16,774::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:35:16,800::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00096109 s, 4.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:35:20,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:35:20,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798454 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:35:26,839::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:35:26,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00136542 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:35:30,183::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:35:30,236::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00225185 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:35:36,908::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:35:36,943::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000861939 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:35:40,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:35:40,292::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013907 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:35:46,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:35:47,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137898 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:35:50,299::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:35:50,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000814146 s, 5.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:35:57,054::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:35:57,097::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137072 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:36:00,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:36:00,395::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00161267 s, 2.5 MB/s\n'; <rc> = 0
>Thread-15147::WARNING::2013-03-18 07:36:03,561::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:36:03,567::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:36:07,137::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:36:07,192::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0143855 s, 285 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:36:10,399::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:36:10,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000796818 s, 5.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:36:13,578::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:36:13,579::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:36:14,014::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:36:14,021::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:36:14,022::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:36:17,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:36:17,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902996 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:36:20,439::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:36:20,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00113153 s, 3.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:36:27,305::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:36:27,341::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000864305 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:36:30,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:36:30,525::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000774191 s, 5.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:36:37,393::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:36:37,422::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00110562 s, 3.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:36:40,530::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:36:40,586::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00223789 s, 1.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:36:47,461::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:36:47,502::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146617 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:36:50,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:36:50,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000948625 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:36:57,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:36:57,574::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000879189 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:37:00,647::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:37:00,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00202034 s, 2.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:37:07,613::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:37:07,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144971 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:37:10,704::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:37:10,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00126707 s, 3.2 MB/s\n'; <rc> = 0
>Thread-15148::WARNING::2013-03-18 07:37:14,051::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:37:14,055::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:37:17,696::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:37:17,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000833644 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:37:20,756::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:37:20,782::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000830836 s, 4.9 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:37:24,060::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:37:24,061::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:37:24,481::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:37:24,488::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:37:24,489::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:37:27,766::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:37:27,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144399 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:37:30,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:37:30,818::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000711347 s, 5.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:37:37,847::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:37:37,892::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00230062 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:37:40,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:37:40,856::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808463 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:37:47,931::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:37:47,972::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137721 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:37:50,863::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:37:50,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141317 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:37:58,011::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:37:58,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137831 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:38:00,911::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:38:00,950::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162455 s, 2.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:38:08,093::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:38:08,132::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147746 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:38:10,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:38:10,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144568 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:38:18,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:38:18,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744096 s, 5.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:38:20,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:38:21,031::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898279 s, 4.6 MB/s\n'; <rc> = 0
>Thread-15149::WARNING::2013-03-18 07:38:24,516::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:38:24,519::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:38:28,245::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:38:28,273::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112876 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:38:31,037::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 07:38:31,038::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 07:38:31,039::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:38:31,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000850216 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:38:31,068::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000850216 s, 2.4 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 07:38:31,068::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 07:38:31,069::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 07:38:31,070::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 07:38:31,071::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:38:31,072::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:38:31,072::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:38:31,073::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:38:31,073::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:38:31,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:38:31,506::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:38:31,511::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:38:31,511::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 07:38:31,514::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 07:38:31,515::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 07:38:31,516::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 07:38:31,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:38:31,871::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:38:31,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:38:31,920::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0191417 s, 214 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:38:34,521::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:38:34,522::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:38:34,948::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:38:34,955::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:38:34,968::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:38:38,312::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 07:38:38,315::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 07:38:38,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:38:38,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000905213 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:38:38,345::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000905213 s, 2.3 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 07:38:38,345::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 07:38:38,346::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 07:38:38,347::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 07:38:38,348::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:38:38,349::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:38:38,350::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:38:38,350::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:38:38,350::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:38:38,351::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:38:38,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:38:38,776::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:38:38,777::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 07:38:38,780::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 07:38:38,782::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 07:38:38,782::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 07:38:38,784::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:38:39,185::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:38:39,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:38:39,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00879551 s, 466 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:38:41,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:38:41,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085295 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:38:49,268::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:38:49,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00078258 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:38:51,958::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:38:51,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150712 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:38:59,337::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:38:59,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147111 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:39:01,993::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:39:02,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150947 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:39:09,415::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:39:09,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138834 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:39:12,027::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:39:12,057::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000851518 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:39:19,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:39:19,518::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00082414 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:39:22,062::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:39:22,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153014 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:39:29,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:39:29,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00225156 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:39:32,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:39:32,153::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00179391 s, 2.3 MB/s\n'; <rc> = 0
>Thread-15150::WARNING::2013-03-18 07:39:35,004::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:39:35,008::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:39:39,638::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:39:39,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000752807 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:39:42,158::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:39:42,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146346 s, 2.8 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:39:45,010::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:39:45,011::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:39:45,469::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:39:45,476::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:39:45,477::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:39:49,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:39:49,730::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000767931 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:39:52,204::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:39:52,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135212 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:39:59,770::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:39:59,803::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00229779 s, 1.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:40:02,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:40:02,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.001985 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:40:09,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:40:09,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00153183 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:40:12,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:40:12,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00085601 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:40:19,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:40:19,963::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00076629 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:40:22,342::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:40:22,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101469 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:40:30,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:40:30,048::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135375 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:40:32,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:40:32,424::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00468774 s, 874 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:40:40,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:40:40,137::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00611048 s, 670 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:40:42,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:40:42,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000837386 s, 4.9 MB/s\n'; <rc> = 0
>Thread-15151::WARNING::2013-03-18 07:40:45,502::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:40:45,508::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:40:50,186::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:40:50,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000795717 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:40:52,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:40:52,513::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00130397 s, 3.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:40:55,509::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 07:40:55,510::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:40:55,511::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:40:55,937::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:40:55,944::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:40:55,945::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:41:00,253::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:41:00,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000910416 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:41:02,518::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:41:02,545::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000761018 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:41:10,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:41:10,353::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000897656 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:41:12,550::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:41:12,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00134103 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:41:20,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:41:20,420::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000808855 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:41:22,597::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:41:22,622::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000924355 s, 4.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:41:30,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:41:30,508::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142552 s, 2.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:41:32,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:41:32,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155343 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:41:40,548::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:41:40,589::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138499 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:41:42,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:41:42,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000858639 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:41:50,628::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:41:50,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159685 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:41:52,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:41:52,764::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135521 s, 3.0 MB/s\n'; <rc> = 0
>Thread-15152::WARNING::2013-03-18 07:41:55,970::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:41:55,976::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:42:00,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:42:00,732::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000932331 s, 4.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:42:02,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:42:02,799::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000813216 s, 5.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:42:05,979::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:42:05,980::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:42:06,415::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:42:06,426::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:42:06,427::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:42:10,773::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:42:10,817::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00148129 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:42:12,804::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:42:12,832::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845655 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:42:20,859::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:42:20,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129122 s, 317 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:42:22,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:42:22,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140561 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:42:30,939::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:42:30,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00100008 s, 4.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:42:32,884::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:42:32,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170662 s, 2.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:42:41,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:42:41,039::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000757703 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:42:42,937::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:42:42,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000803055 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:42:51,078::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:42:51,122::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175433 s, 2.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:42:52,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:42:53,012::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0014776 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:43:01,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:43:01,205::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157228 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:43:03,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:43:03,052::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000815497 s, 5.0 MB/s\n'; <rc> = 0
>Thread-15153::WARNING::2013-03-18 07:43:06,459::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:43:06,463::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:43:11,249::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:43:11,275::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845714 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:43:13,058::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:43:13,085::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101642 s, 4.0 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:43:16,465::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:43:16,466::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:43:16,925::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:43:16,931::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:43:16,932::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:43:21,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:43:21,349::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079486 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:43:23,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:43:23,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140058 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:43:31,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:43:31,424::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00190464 s, 2.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:43:33,142::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 07:43:33,143::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 07:43:33,144::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:43:33,171::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0008284 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:43:33,172::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0008284 s, 2.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 07:43:33,173::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 07:43:33,173::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 07:43:33,175::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 07:43:33,176::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:43:33,176::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:43:33,177::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:43:33,177::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:43:33,177::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:43:33,178::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:43:33,570::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:43:33,574::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:43:33,575::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 07:43:33,577::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 07:43:33,578::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 07:43:33,579::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 07:43:33,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:43:33,959::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:43:33,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:43:34,022::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123727 s, 331 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:43:41,462::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 07:43:41,464::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 07:43:41,465::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:43:41,493::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000918941 s, 2.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:43:41,494::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000918941 s, 2.2 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 07:43:41,495::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 07:43:41,495::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 07:43:41,496::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 07:43:41,497::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:43:41,498::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:43:41,498::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:43:41,499::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:43:41,499::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:43:41,500::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:43:41,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:43:41,921::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:43:41,922::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 07:43:41,925::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 07:43:41,926::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 07:43:41,927::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 07:43:41,928::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:43:42,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:43:42,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:43:42,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0100775 s, 406 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:43:44,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:43:44,077::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00149749 s, 2.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:43:52,361::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:43:52,408::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013877 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:43:54,082::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:43:54,115::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000791632 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:44:02,450::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:44:02,481::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00104551 s, 3.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:44:04,122::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:44:04,161::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133344 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:44:12,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:44:12,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00128712 s, 3.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:44:14,166::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:44:14,214::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780983 s, 5.2 MB/s\n'; <rc> = 0
>Thread-15154::WARNING::2013-03-18 07:44:16,957::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:44:16,960::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:44:22,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:44:22,656::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000854072 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:44:24,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:44:24,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000792664 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:44:26,963::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:44:26,964::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:44:27,399::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:44:27,406::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:44:27,407::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:44:32,699::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:44:32,724::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000818714 s, 5.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:44:34,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:44:34,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159397 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:44:42,763::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:44:42,790::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000722655 s, 5.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:44:44,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:44:44,358::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0101376 s, 404 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:44:52,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:44:52,906::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00505938 s, 810 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:44:54,364::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:44:54,401::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794407 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:45:02,945::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:45:02,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137604 s, 3.0 MB/s\n'; <rc> = 0
>Thread-15156::DEBUG::2013-03-18 07:45:04,327::BindingXMLRPC::903::vds::(wrapper) client [10.34.1.147]::call getCapabilities with () {}
>Thread-15156::DEBUG::2013-03-18 07:45:04,377::BindingXMLRPC::910::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:38761586c140'}], 'FC': []}, 'packages2': {'kernel': {'release': '358.2.1.el6.x86_64', 'buildtime': 1361362657.0, 'version': '2.6.32'}, 'spice-server': {'release': '12.el6', 'buildtime': 1358331735L, 'version': '0.12.0'}, 'vdsm': {'release': '1.8.el6ev', 'buildtime': 1362555435L, 'version': '4.10.2'}, 'qemu-kvm': {'release': '2.355.el6_4.2', 'buildtime': 1362482745L, 'version': '0.12.1.2'}, 'libvirt': {'release': '18.el6', 'buildtime': 1359405942L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.355.el6_4.2', 'buildtime': 1362482745L, 'version': '0.12.1.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 Duo CPU     T7700  @ 2.40GHz', 'hooks': {'before_vm_start': {'50_vhostmd': {'md5': 'f3ee6dbf6fbd01333bd3e32afec4fbba'}}, 'after_vm_destroy': {'50_vhostmd': {'md5': '5662659ddc999e23946a1f6cf7a73796'}}, 'before_vm_dehibernate': {'50_vhostmd': {'md5': 'f3ee6dbf6fbd01333bd3e32afec4fbba'}}, 'before_vm_migrate_destination': {'50_vhostmd': {'md5': 'f3ee6dbf6fbd01333bd3e32afec4fbba'}}}, 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.1.137', 'cfg': {'IPV6FORWARDING': 'no', 'IPV6INIT': 'no', 'IPV6_AUTOCONF': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'PEERNTP': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'gateway': '10.34.1.254', 'ports': ['eth0']}}, 'bridges': {'rhevm': {'addr': '10.34.1.137', 'cfg': {'IPV6FORWARDING': 'no', 'IPV6INIT': 'no', 'IPV6_AUTOCONF': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'PEERNTP': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ports': ['eth0']}}, 'uuid': 'FD1A802B-6FFF-2241-1C2D-2D8A4FF68C9B_52:54:00:11:9f:24', 'lastClientIface': 'rhevm', 'nics': {'eth0': {'addr': '', 'cfg': {'IPV6FORWARDING': 'no', 'BRIDGE': 'rhevm', 'IPV6INIT': 'no', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'no', 'DELAY': '0', 'HWADDR': '52:54:00:11:9f:24', 'DEVICE': 'eth0', 'PEERNTP': 'yes', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '', 'hwaddr': '52:54:00:11:9f:24', 'speed': 0}}, 'software_revision': '1.8', 'management_ip': '', 'clusterLevels': ['3.0', '3.1'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,mmx,fxsr,sse,sse2,ss,ht,syscall,nx,lm,constant_tsc,arch_perfmon,rep_good,unfair_spinlock,pni,vmx,ssse3,cx16,hypervisor,lahf_lm,model_Conroe', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:38761586c140', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1'], 'reservedMem': '321', 'bondings': {'bond4': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond0': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond1': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond2': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond3': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}}, 'software_version': '4.10', 'memSize': '1831', 'cpuSpeed': '2133.408', 'cpuSockets': '1', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'version_name': 'Snow Man', 'emulatedMachines': [u'rhel6.4.0', u'pc', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '20130306.2.el6_4', 'version': '6.4', 'name': 'RHEV Hypervisor'}, 'lastClient': '10.34.1.147'}}
>Thread-22::DEBUG::2013-03-18 07:45:04,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:45:04,465::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111336 s, 3.7 MB/s\n'; <rc> = 0
>Thread-15157::DEBUG::2013-03-18 07:45:04,676::BindingXMLRPC::903::vds::(wrapper) client [10.34.1.147]::call getCapabilities with () {}
>Thread-15157::DEBUG::2013-03-18 07:45:04,730::BindingXMLRPC::910::vds::(wrapper) return getCapabilities with {'status': {'message': 'Done', 'code': 0}, 'info': {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:38761586c140'}], 'FC': []}, 'packages2': {'kernel': {'release': '358.2.1.el6.x86_64', 'buildtime': 1361362657.0, 'version': '2.6.32'}, 'spice-server': {'release': '12.el6', 'buildtime': 1358331735L, 'version': '0.12.0'}, 'vdsm': {'release': '1.8.el6ev', 'buildtime': 1362555435L, 'version': '4.10.2'}, 'qemu-kvm': {'release': '2.355.el6_4.2', 'buildtime': 1362482745L, 'version': '0.12.1.2'}, 'libvirt': {'release': '18.el6', 'buildtime': 1359405942L, 'version': '0.10.2'}, 'qemu-img': {'release': '2.355.el6_4.2', 'buildtime': 1362482745L, 'version': '0.12.1.2'}}, 'cpuModel': 'Intel(R) Core(TM)2 Duo CPU     T7700  @ 2.40GHz', 'hooks': {'before_vm_start': {'50_vhostmd': {'md5': 'f3ee6dbf6fbd01333bd3e32afec4fbba'}}, 'after_vm_destroy': {'50_vhostmd': {'md5': '5662659ddc999e23946a1f6cf7a73796'}}, 'before_vm_dehibernate': {'50_vhostmd': {'md5': 'f3ee6dbf6fbd01333bd3e32afec4fbba'}}, 'before_vm_migrate_destination': {'50_vhostmd': {'md5': 'f3ee6dbf6fbd01333bd3e32afec4fbba'}}}, 'vmTypes': ['kvm'], 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.1.137', 'cfg': {'IPV6FORWARDING': 'no', 'IPV6INIT': 'no', 'IPV6_AUTOCONF': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'PEERNTP': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'bridged': True, 'gateway': '10.34.1.254', 'ports': ['eth0']}}, 'bridges': {'rhevm': {'addr': '10.34.1.137', 'cfg': {'IPV6FORWARDING': 'no', 'IPV6INIT': 'no', 'IPV6_AUTOCONF': 'no', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'DEVICE': 'rhevm', 'PEERNTP': 'yes', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '255.255.255.0', 'stp': 'off', 'ports': ['eth0']}}, 'uuid': 'FD1A802B-6FFF-2241-1C2D-2D8A4FF68C9B_52:54:00:11:9f:24', 'lastClientIface': 'rhevm', 'nics': {'eth0': {'addr': '', 'cfg': {'IPV6FORWARDING': 'no', 'BRIDGE': 'rhevm', 'IPV6INIT': 'no', 'NM_CONTROLLED': 'no', 'IPV6_AUTOCONF': 'no', 'DELAY': '0', 'HWADDR': '52:54:00:11:9f:24', 'DEVICE': 'eth0', 'PEERNTP': 'yes', 'ONBOOT': 'yes'}, 'mtu': '1500', 'netmask': '', 'hwaddr': '52:54:00:11:9f:24', 'speed': 0}}, 'software_revision': '1.8', 'management_ip': '', 'clusterLevels': ['3.0', '3.1'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,mmx,fxsr,sse,sse2,ss,ht,syscall,nx,lm,constant_tsc,arch_perfmon,rep_good,unfair_spinlock,pni,vmx,ssse3,cx16,hypervisor,lahf_lm,model_Conroe', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:38761586c140', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1'], 'reservedMem': '321', 'bondings': {'bond4': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond0': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond1': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond2': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}, 'bond3': {'addr': '', 'cfg': {}, 'mtu': '1500', 'netmask': '', 'slaves': [], 'hwaddr': '00:00:00:00:00:00'}}, 'software_version': '4.10', 'memSize': '1831', 'cpuSpeed': '2133.408', 'cpuSockets': '1', 'vlans': {}, 'cpuCores': '2', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'version_name': 'Snow Man', 'emulatedMachines': [u'rhel6.4.0', u'pc', u'rhel6.3.0', u'rhel6.2.0', u'rhel6.1.0', u'rhel6.0.0', u'rhel5.5.0', u'rhel5.4.4', u'rhel5.4.0'], 'operatingSystem': {'release': '20130306.2.el6_4', 'version': '6.4', 'name': 'RHEV Hypervisor'}, 'lastClient': '10.34.1.147'}}
>Thread-15159::DEBUG::2013-03-18 07:45:05,627::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15159::DEBUG::2013-03-18 07:45:05,629::task::568::TaskManager.Task::(_updateState) Task=`d6121649-c9f1-4098-ae66-34573a8ba108`::moving from state init -> state preparing
>Thread-15159::INFO::2013-03-18 07:45:05,630::logUtils::37::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=3, spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', conList=[{'connection': '10.34.1.214', 'iqn': 'iqn.2003-01.org.linux-iscsi.dhcp-1-214.x8664:sn.113e745f4931', 'portal': '1', 'user': '', 'password': '******', 'id': '76e6e082-bc2a-47cb-8062-fde36badf398', 'port': '3260'}], options=None)
>Thread-15159::DEBUG::2013-03-18 07:45:05,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/iscsiadm -m node -T iqn.2003-01.org.linux-iscsi.dhcp-1-214.x8664:sn.113e745f4931 -I default -p 10.34.1.214:3260 --op=new' (cwd None)
>Thread-15159::DEBUG::2013-03-18 07:45:05,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-15159::DEBUG::2013-03-18 07:45:05,690::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/iscsiadm -m node -T iqn.2003-01.org.linux-iscsi.dhcp-1-214.x8664:sn.113e745f4931 -I default -p 10.34.1.214:3260 -l' (cwd None)
>Thread-15159::DEBUG::2013-03-18 07:45:05,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-15159::DEBUG::2013-03-18 07:45:05,754::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/iscsiadm -m node -T iqn.2003-01.org.linux-iscsi.dhcp-1-214.x8664:sn.113e745f4931 -I default -p 10.34.1.214:3260 -n node.startup -v manual --op=update' (cwd None)
>Thread-15159::DEBUG::2013-03-18 07:45:05,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-15159::INFO::2013-03-18 07:45:05,825::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '76e6e082-bc2a-47cb-8062-fde36badf398'}]}
>Thread-15159::DEBUG::2013-03-18 07:45:05,826::task::1151::TaskManager.Task::(prepare) Task=`d6121649-c9f1-4098-ae66-34573a8ba108`::finished: {'statuslist': [{'status': 0, 'id': '76e6e082-bc2a-47cb-8062-fde36badf398'}]}
>Thread-15159::DEBUG::2013-03-18 07:45:05,826::task::568::TaskManager.Task::(_updateState) Task=`d6121649-c9f1-4098-ae66-34573a8ba108`::moving from state preparing -> state finished
>Thread-15159::DEBUG::2013-03-18 07:45:05,827::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15159::DEBUG::2013-03-18 07:45:05,827::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15159::DEBUG::2013-03-18 07:45:05,827::task::957::TaskManager.Task::(_decref) Task=`d6121649-c9f1-4098-ae66-34573a8ba108`::ref 0 aborting False
>Thread-15160::DEBUG::2013-03-18 07:45:05,877::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15160::DEBUG::2013-03-18 07:45:05,879::task::568::TaskManager.Task::(_updateState) Task=`6796fa4a-e49e-4262-b6da-80de2609d2c5`::moving from state init -> state preparing
>Thread-15160::INFO::2013-03-18 07:45:05,879::logUtils::37::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', conList=[{'connection': '10.34.1.147:/rhev/isos', 'iqn': '', 'portal': '', 'user': '', 'password': '******', 'id': '7b4a1a9e-a2a0-4849-8ab1-8fbada548651', 'port': ''}], options=None)
>Thread-15160::INFO::2013-03-18 07:45:05,910::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '7b4a1a9e-a2a0-4849-8ab1-8fbada548651'}]}
>Thread-15160::DEBUG::2013-03-18 07:45:05,911::task::1151::TaskManager.Task::(prepare) Task=`6796fa4a-e49e-4262-b6da-80de2609d2c5`::finished: {'statuslist': [{'status': 0, 'id': '7b4a1a9e-a2a0-4849-8ab1-8fbada548651'}]}
>Thread-15160::DEBUG::2013-03-18 07:45:05,911::task::568::TaskManager.Task::(_updateState) Task=`6796fa4a-e49e-4262-b6da-80de2609d2c5`::moving from state preparing -> state finished
>Thread-15160::DEBUG::2013-03-18 07:45:05,911::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15160::DEBUG::2013-03-18 07:45:05,911::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15160::DEBUG::2013-03-18 07:45:05,912::task::957::TaskManager.Task::(_decref) Task=`6796fa4a-e49e-4262-b6da-80de2609d2c5`::ref 0 aborting False
>Thread-15161::DEBUG::2013-03-18 07:45:06,008::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15161::DEBUG::2013-03-18 07:45:06,009::task::568::TaskManager.Task::(_updateState) Task=`2cd5ea40-1a1f-4672-8090-1255f9b8730a`::moving from state init -> state preparing
>Thread-15161::INFO::2013-03-18 07:45:06,010::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', hostID=2, scsiKey='a1112cc0-b4af-4396-a3b6-2a3916293a6b', msdUUID='4bf968ef-020a-47ee-a1ae-f85eaeb8751a', masterVersion=1, options=None)
>Thread-15161::DEBUG::2013-03-18 07:45:06,013::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`469e90f4-14bf-4de0-953d-5f839b96ae95`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15161::DEBUG::2013-03-18 07:45:06,013::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-15161::DEBUG::2013-03-18 07:45:06,014::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15161::DEBUG::2013-03-18 07:45:06,014::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`469e90f4-14bf-4de0-953d-5f839b96ae95`::Granted request
>Thread-15161::DEBUG::2013-03-18 07:45:06,016::sp::1536::Storage.StoragePool::(getMasterDomain) Master domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a verified, version 1
>Thread-15161::DEBUG::2013-03-18 07:45:06,017::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15161::DEBUG::2013-03-18 07:45:06,017::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15161::DEBUG::2013-03-18 07:45:06,017::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15161::DEBUG::2013-03-18 07:45:06,017::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15161::INFO::2013-03-18 07:45:06,018::logUtils::39::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: None
>Thread-15161::DEBUG::2013-03-18 07:45:06,018::task::1151::TaskManager.Task::(prepare) Task=`2cd5ea40-1a1f-4672-8090-1255f9b8730a`::finished: None
>Thread-15161::DEBUG::2013-03-18 07:45:06,018::task::568::TaskManager.Task::(_updateState) Task=`2cd5ea40-1a1f-4672-8090-1255f9b8730a`::moving from state preparing -> state finished
>Thread-15161::DEBUG::2013-03-18 07:45:06,019::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15161::DEBUG::2013-03-18 07:45:06,019::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15161::DEBUG::2013-03-18 07:45:06,019::task::957::TaskManager.Task::(_decref) Task=`2cd5ea40-1a1f-4672-8090-1255f9b8730a`::ref 0 aborting False
>Thread-15162::DEBUG::2013-03-18 07:45:06,051::task::568::TaskManager.Task::(_updateState) Task=`6d3b032f-7f18-459a-8c4b-8162d38dac80`::moving from state init -> state preparing
>Thread-15162::INFO::2013-03-18 07:45:06,052::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15162::INFO::2013-03-18 07:45:06,053::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0496990680695', 'lastCheck': '3.0', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '49.1', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0601680278778', 'lastCheck': '1.6', 'code': 0, 'valid': True}}
>Thread-15162::DEBUG::2013-03-18 07:45:06,053::task::1151::TaskManager.Task::(prepare) Task=`6d3b032f-7f18-459a-8c4b-8162d38dac80`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0496990680695', 'lastCheck': '3.0', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '49.1', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0601680278778', 'lastCheck': '1.6', 'code': 0, 'valid': True}}
>Thread-15162::DEBUG::2013-03-18 07:45:06,053::task::568::TaskManager.Task::(_updateState) Task=`6d3b032f-7f18-459a-8c4b-8162d38dac80`::moving from state preparing -> state finished
>Thread-15162::DEBUG::2013-03-18 07:45:06,053::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15162::DEBUG::2013-03-18 07:45:06,054::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15162::DEBUG::2013-03-18 07:45:06,054::task::957::TaskManager.Task::(_decref) Task=`6d3b032f-7f18-459a-8c4b-8162d38dac80`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:45:13,034::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:45:13,100::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0226455 s, 181 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:45:14,469::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:45:14,531::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142261 s, 2.9 MB/s\n'; <rc> = 0
>Thread-15167::DEBUG::2013-03-18 07:45:19,369::task::568::TaskManager.Task::(_updateState) Task=`0a758dba-424a-4ea3-9165-f4ae4d812c1f`::moving from state init -> state preparing
>Thread-15167::INFO::2013-03-18 07:45:19,370::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15167::INFO::2013-03-18 07:45:19,371::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0672869682312', 'lastCheck': '6.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '62.4', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0633227825165', 'lastCheck': '4.8', 'code': 0, 'valid': True}}
>Thread-15167::DEBUG::2013-03-18 07:45:19,371::task::1151::TaskManager.Task::(prepare) Task=`0a758dba-424a-4ea3-9165-f4ae4d812c1f`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0672869682312', 'lastCheck': '6.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '62.4', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0633227825165', 'lastCheck': '4.8', 'code': 0, 'valid': True}}
>Thread-15167::DEBUG::2013-03-18 07:45:19,371::task::568::TaskManager.Task::(_updateState) Task=`0a758dba-424a-4ea3-9165-f4ae4d812c1f`::moving from state preparing -> state finished
>Thread-15167::DEBUG::2013-03-18 07:45:19,372::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15167::DEBUG::2013-03-18 07:45:19,372::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15167::DEBUG::2013-03-18 07:45:19,372::task::957::TaskManager.Task::(_decref) Task=`0a758dba-424a-4ea3-9165-f4ae4d812c1f`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:45:23,140::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:45:23,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0015985 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:45:24,538::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:45:24,595::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185021 s, 221 kB/s\n'; <rc> = 0
>Thread-15155::WARNING::2013-03-18 07:45:27,442::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:45:27,446::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-15173::DEBUG::2013-03-18 07:45:29,003::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15173::DEBUG::2013-03-18 07:45:29,004::task::568::TaskManager.Task::(_updateState) Task=`3e5c7b16-69d6-4353-9cb4-1128ba7d8f9e`::moving from state init -> state preparing
>Thread-15173::INFO::2013-03-18 07:45:29,005::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15173::INFO::2013-03-18 07:45:29,005::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15173::DEBUG::2013-03-18 07:45:29,006::task::1151::TaskManager.Task::(prepare) Task=`3e5c7b16-69d6-4353-9cb4-1128ba7d8f9e`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15173::DEBUG::2013-03-18 07:45:29,006::task::568::TaskManager.Task::(_updateState) Task=`3e5c7b16-69d6-4353-9cb4-1128ba7d8f9e`::moving from state preparing -> state finished
>Thread-15173::DEBUG::2013-03-18 07:45:29,006::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15173::DEBUG::2013-03-18 07:45:29,006::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15173::DEBUG::2013-03-18 07:45:29,007::task::957::TaskManager.Task::(_decref) Task=`3e5c7b16-69d6-4353-9cb4-1128ba7d8f9e`::ref 0 aborting False
>Thread-15174::DEBUG::2013-03-18 07:45:29,806::task::568::TaskManager.Task::(_updateState) Task=`cea89ab5-3cdc-4c51-a6e5-b64c8322b1f7`::moving from state init -> state preparing
>Thread-15174::INFO::2013-03-18 07:45:29,807::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15174::INFO::2013-03-18 07:45:29,808::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0426859855652', 'lastCheck': '6.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '2.4', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0615568161011', 'lastCheck': '5.2', 'code': 0, 'valid': True}}
>Thread-15174::DEBUG::2013-03-18 07:45:29,808::task::1151::TaskManager.Task::(prepare) Task=`cea89ab5-3cdc-4c51-a6e5-b64c8322b1f7`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0426859855652', 'lastCheck': '6.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '2.4', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0615568161011', 'lastCheck': '5.2', 'code': 0, 'valid': True}}
>Thread-15174::DEBUG::2013-03-18 07:45:29,809::task::568::TaskManager.Task::(_updateState) Task=`cea89ab5-3cdc-4c51-a6e5-b64c8322b1f7`::moving from state preparing -> state finished
>Thread-15174::DEBUG::2013-03-18 07:45:29,809::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15174::DEBUG::2013-03-18 07:45:29,809::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15174::DEBUG::2013-03-18 07:45:29,810::task::957::TaskManager.Task::(_decref) Task=`cea89ab5-3cdc-4c51-a6e5-b64c8322b1f7`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:45:33,221::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:45:33,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0242284 s, 169 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:45:34,607::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:45:34,660::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132786 s, 3.1 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:45:37,447::misc::1025::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-21::DEBUG::2013-03-18 07:45:37,448::misc::1027::SamplingMethod::(__call__) Got in to sampling method
>Thread-21::DEBUG::2013-03-18 07:45:37,449::misc::1025::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-21::DEBUG::2013-03-18 07:45:37,449::misc::1027::SamplingMethod::(__call__) Got in to sampling method
>Thread-21::DEBUG::2013-03-18 07:45:37,449::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:45:37,505::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:45:37,507::misc::1035::SamplingMethod::(__call__) Returning last result
>MainProcess|Thread-21::DEBUG::2013-03-18 07:45:37,520::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd of=/sys/class/scsi_host/host0/scan' (cwd None)
>MainProcess|Thread-21::DEBUG::2013-03-18 07:45:38,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd of=/sys/class/scsi_host/host1/scan' (cwd None)
>Thread-15180::DEBUG::2013-03-18 07:45:39,106::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15180::DEBUG::2013-03-18 07:45:39,108::task::568::TaskManager.Task::(_updateState) Task=`7565da55-d38a-47f4-85c6-59647baaac34`::moving from state init -> state preparing
>Thread-15180::INFO::2013-03-18 07:45:39,109::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15180::INFO::2013-03-18 07:45:39,110::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15180::DEBUG::2013-03-18 07:45:39,110::task::1151::TaskManager.Task::(prepare) Task=`7565da55-d38a-47f4-85c6-59647baaac34`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15180::DEBUG::2013-03-18 07:45:39,110::task::568::TaskManager.Task::(_updateState) Task=`7565da55-d38a-47f4-85c6-59647baaac34`::moving from state preparing -> state finished
>Thread-15180::DEBUG::2013-03-18 07:45:39,111::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15180::DEBUG::2013-03-18 07:45:39,111::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15180::DEBUG::2013-03-18 07:45:39,111::task::957::TaskManager.Task::(_decref) Task=`7565da55-d38a-47f4-85c6-59647baaac34`::ref 0 aborting False
>MainProcess|Thread-21::DEBUG::2013-03-18 07:45:39,557::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd of=/sys/class/scsi_host/host2/scan' (cwd None)
>MainProcess|Thread-21::DEBUG::2013-03-18 07:45:39,570::iscsi::388::Storage.ISCSI::(forceIScsiScan) Performing SCSI scan, this will take up to 30 seconds
>Thread-15181::DEBUG::2013-03-18 07:45:40,171::task::568::TaskManager.Task::(_updateState) Task=`b9e18bac-1cc4-4cca-a144-b6fe4b4c806b`::moving from state init -> state preparing
>Thread-15181::INFO::2013-03-18 07:45:40,171::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15181::INFO::2013-03-18 07:45:40,175::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0702550411224', 'lastCheck': '6.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '12.7', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0550339221954', 'lastCheck': '5.5', 'code': 0, 'valid': True}}
>Thread-15181::DEBUG::2013-03-18 07:45:40,175::task::1151::TaskManager.Task::(prepare) Task=`b9e18bac-1cc4-4cca-a144-b6fe4b4c806b`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0702550411224', 'lastCheck': '6.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '12.7', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0550339221954', 'lastCheck': '5.5', 'code': 0, 'valid': True}}
>Thread-15181::DEBUG::2013-03-18 07:45:40,175::task::568::TaskManager.Task::(_updateState) Task=`b9e18bac-1cc4-4cca-a144-b6fe4b4c806b`::moving from state preparing -> state finished
>Thread-15181::DEBUG::2013-03-18 07:45:40,176::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15181::DEBUG::2013-03-18 07:45:40,177::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15181::DEBUG::2013-03-18 07:45:40,178::task::957::TaskManager.Task::(_decref) Task=`b9e18bac-1cc4-4cca-a144-b6fe4b4c806b`::ref 0 aborting False
>Thread-21::DEBUG::2013-03-18 07:45:41,576::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:45:41,680::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:45:41,682::lvm::482::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:45:41,683::lvm::484::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-21::DEBUG::2013-03-18 07:45:41,684::lvm::493::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:45:41,685::lvm::495::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-21::DEBUG::2013-03-18 07:45:41,685::lvm::513::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:45:41,686::lvm::515::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-21::DEBUG::2013-03-18 07:45:41,686::misc::1035::SamplingMethod::(__call__) Returning last result
>Thread-21::DEBUG::2013-03-18 07:45:41,687::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:45:41,693::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:45:42,127::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:45:42,133::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:45:42,135::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:45:43,331::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:45:43,333::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:45:43,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:45:43,813::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:45:43,814::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:45:43,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00112352 s, 3.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:45:44,665::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:45:44,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:45:45,037::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:45:45,040::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:45:45,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:45:45,070::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000744897 s, 5.5 MB/s\n'; <rc> = 0
>Thread-15188::DEBUG::2013-03-18 07:45:50,549::task::568::TaskManager.Task::(_updateState) Task=`59101b73-6e53-4db8-8c58-40e26596eece`::moving from state init -> state preparing
>Thread-15188::INFO::2013-03-18 07:45:50,550::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15188::INFO::2013-03-18 07:45:50,551::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0343029499054', 'lastCheck': '6.7', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '23.1', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0300409793854', 'lastCheck': '5.5', 'code': 0, 'valid': True}}
>Thread-15188::DEBUG::2013-03-18 07:45:50,551::task::1151::TaskManager.Task::(prepare) Task=`59101b73-6e53-4db8-8c58-40e26596eece`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0343029499054', 'lastCheck': '6.7', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '23.1', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0300409793854', 'lastCheck': '5.5', 'code': 0, 'valid': True}}
>Thread-15188::DEBUG::2013-03-18 07:45:50,551::task::568::TaskManager.Task::(_updateState) Task=`59101b73-6e53-4db8-8c58-40e26596eece`::moving from state preparing -> state finished
>Thread-15188::DEBUG::2013-03-18 07:45:50,551::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15188::DEBUG::2013-03-18 07:45:50,552::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15188::DEBUG::2013-03-18 07:45:50,552::task::957::TaskManager.Task::(_decref) Task=`59101b73-6e53-4db8-8c58-40e26596eece`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:45:53,886::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:45:53,912::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087785 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:45:55,075::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:45:55,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142804 s, 2.9 MB/s\n'; <rc> = 0
>Thread-15194::DEBUG::2013-03-18 07:46:00,954::task::568::TaskManager.Task::(_updateState) Task=`80ed8f9a-1a17-4d41-a689-60f720cf74fb`::moving from state init -> state preparing
>Thread-15194::INFO::2013-03-18 07:46:00,955::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15194::INFO::2013-03-18 07:46:00,956::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0274350643158', 'lastCheck': '7.0', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '33.5', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0371181964874', 'lastCheck': '5.8', 'code': 0, 'valid': True}}
>Thread-15194::DEBUG::2013-03-18 07:46:00,956::task::1151::TaskManager.Task::(prepare) Task=`80ed8f9a-1a17-4d41-a689-60f720cf74fb`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0274350643158', 'lastCheck': '7.0', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '33.5', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0371181964874', 'lastCheck': '5.8', 'code': 0, 'valid': True}}
>Thread-15194::DEBUG::2013-03-18 07:46:00,956::task::568::TaskManager.Task::(_updateState) Task=`80ed8f9a-1a17-4d41-a689-60f720cf74fb`::moving from state preparing -> state finished
>Thread-15194::DEBUG::2013-03-18 07:46:00,956::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15194::DEBUG::2013-03-18 07:46:00,957::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15194::DEBUG::2013-03-18 07:46:00,957::task::957::TaskManager.Task::(_decref) Task=`80ed8f9a-1a17-4d41-a689-60f720cf74fb`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:46:03,957::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:46:03,991::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160377 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:46:05,116::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:46:05,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000880734 s, 4.7 MB/s\n'; <rc> = 0
>Thread-15200::DEBUG::2013-03-18 07:46:11,366::task::568::TaskManager.Task::(_updateState) Task=`7513cccc-1e61-41cb-a291-3c3834024791`::moving from state init -> state preparing
>Thread-15200::INFO::2013-03-18 07:46:11,367::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15200::INFO::2013-03-18 07:46:11,368::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0365149974823', 'lastCheck': '7.3', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '43.9', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0310790538788', 'lastCheck': '6.2', 'code': 0, 'valid': True}}
>Thread-15200::DEBUG::2013-03-18 07:46:11,368::task::1151::TaskManager.Task::(prepare) Task=`7513cccc-1e61-41cb-a291-3c3834024791`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0365149974823', 'lastCheck': '7.3', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '43.9', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0310790538788', 'lastCheck': '6.2', 'code': 0, 'valid': True}}
>Thread-15200::DEBUG::2013-03-18 07:46:11,368::task::568::TaskManager.Task::(_updateState) Task=`7513cccc-1e61-41cb-a291-3c3834024791`::moving from state preparing -> state finished
>Thread-15200::DEBUG::2013-03-18 07:46:11,369::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15200::DEBUG::2013-03-18 07:46:11,369::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15200::DEBUG::2013-03-18 07:46:11,369::task::957::TaskManager.Task::(_decref) Task=`7513cccc-1e61-41cb-a291-3c3834024791`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:46:14,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:46:14,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000836358 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:46:15,163::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:46:15,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00137433 s, 3.0 MB/s\n'; <rc> = 0
>Thread-15205::DEBUG::2013-03-18 07:46:19,402::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15205::DEBUG::2013-03-18 07:46:19,404::task::568::TaskManager.Task::(_updateState) Task=`07c1590e-56f5-4cc6-aff8-d1f698ff9d8a`::moving from state init -> state preparing
>Thread-15205::INFO::2013-03-18 07:46:19,407::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15205::INFO::2013-03-18 07:46:19,408::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15205::DEBUG::2013-03-18 07:46:19,409::task::1151::TaskManager.Task::(prepare) Task=`07c1590e-56f5-4cc6-aff8-d1f698ff9d8a`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15205::DEBUG::2013-03-18 07:46:19,409::task::568::TaskManager.Task::(_updateState) Task=`07c1590e-56f5-4cc6-aff8-d1f698ff9d8a`::moving from state preparing -> state finished
>Thread-15205::DEBUG::2013-03-18 07:46:19,409::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15205::DEBUG::2013-03-18 07:46:19,410::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15205::DEBUG::2013-03-18 07:46:19,410::task::957::TaskManager.Task::(_decref) Task=`07c1590e-56f5-4cc6-aff8-d1f698ff9d8a`::ref 0 aborting False
>Thread-15207::DEBUG::2013-03-18 07:46:21,695::task::568::TaskManager.Task::(_updateState) Task=`29833b82-543b-44fe-9a43-1dae88f56a1a`::moving from state init -> state preparing
>Thread-15207::INFO::2013-03-18 07:46:21,695::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15207::INFO::2013-03-18 07:46:21,696::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0367391109467', 'lastCheck': '7.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '54.2', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0374500751495', 'lastCheck': '6.5', 'code': 0, 'valid': True}}
>Thread-15207::DEBUG::2013-03-18 07:46:21,696::task::1151::TaskManager.Task::(prepare) Task=`29833b82-543b-44fe-9a43-1dae88f56a1a`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0367391109467', 'lastCheck': '7.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '54.2', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0374500751495', 'lastCheck': '6.5', 'code': 0, 'valid': True}}
>Thread-15207::DEBUG::2013-03-18 07:46:21,697::task::568::TaskManager.Task::(_updateState) Task=`29833b82-543b-44fe-9a43-1dae88f56a1a`::moving from state preparing -> state finished
>Thread-15207::DEBUG::2013-03-18 07:46:21,697::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15207::DEBUG::2013-03-18 07:46:21,697::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15207::DEBUG::2013-03-18 07:46:21,697::task::957::TaskManager.Task::(_decref) Task=`29833b82-543b-44fe-9a43-1dae88f56a1a`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:46:24,109::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:46:24,143::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00162843 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:46:25,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:46:25,242::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00315966 s, 1.3 MB/s\n'; <rc> = 0
>Thread-15212::DEBUG::2013-03-18 07:46:29,438::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15212::DEBUG::2013-03-18 07:46:29,440::task::568::TaskManager.Task::(_updateState) Task=`c1e230c4-f0b2-4155-84b2-11d538f412f3`::moving from state init -> state preparing
>Thread-15212::INFO::2013-03-18 07:46:29,440::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15212::INFO::2013-03-18 07:46:29,441::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15212::DEBUG::2013-03-18 07:46:29,441::task::1151::TaskManager.Task::(prepare) Task=`c1e230c4-f0b2-4155-84b2-11d538f412f3`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15212::DEBUG::2013-03-18 07:46:29,442::task::568::TaskManager.Task::(_updateState) Task=`c1e230c4-f0b2-4155-84b2-11d538f412f3`::moving from state preparing -> state finished
>Thread-15212::DEBUG::2013-03-18 07:46:29,442::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15212::DEBUG::2013-03-18 07:46:29,442::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15212::DEBUG::2013-03-18 07:46:29,442::task::957::TaskManager.Task::(_decref) Task=`c1e230c4-f0b2-4155-84b2-11d538f412f3`::ref 0 aborting False
>Thread-15214::DEBUG::2013-03-18 07:46:32,055::task::568::TaskManager.Task::(_updateState) Task=`3fa39529-06e6-4c07-9e59-c99f35b3d13b`::moving from state init -> state preparing
>Thread-15214::INFO::2013-03-18 07:46:32,056::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15214::INFO::2013-03-18 07:46:32,057::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0354771614075', 'lastCheck': '7.9', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '64.6', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0379021167755', 'lastCheck': '6.8', 'code': 0, 'valid': True}}
>Thread-15214::DEBUG::2013-03-18 07:46:32,057::task::1151::TaskManager.Task::(prepare) Task=`3fa39529-06e6-4c07-9e59-c99f35b3d13b`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0354771614075', 'lastCheck': '7.9', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '64.6', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0379021167755', 'lastCheck': '6.8', 'code': 0, 'valid': True}}
>Thread-15214::DEBUG::2013-03-18 07:46:32,057::task::568::TaskManager.Task::(_updateState) Task=`3fa39529-06e6-4c07-9e59-c99f35b3d13b`::moving from state preparing -> state finished
>Thread-15214::DEBUG::2013-03-18 07:46:32,057::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15214::DEBUG::2013-03-18 07:46:32,058::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15214::DEBUG::2013-03-18 07:46:32,058::task::957::TaskManager.Task::(_decref) Task=`3fa39529-06e6-4c07-9e59-c99f35b3d13b`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:46:34,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:46:34,237::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00382591 s, 1.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:46:35,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:46:35,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00250148 s, 1.6 MB/s\n'; <rc> = 0
>Thread-15183::WARNING::2013-03-18 07:46:42,163::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:46:42,166::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-15220::DEBUG::2013-03-18 07:46:42,441::task::568::TaskManager.Task::(_updateState) Task=`96bc048f-364c-4c7b-ab48-df1cfd73c0f3`::moving from state init -> state preparing
>Thread-15220::INFO::2013-03-18 07:46:42,442::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15220::INFO::2013-03-18 07:46:42,443::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0567080974579', 'lastCheck': '8.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '0.3', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0394651889801', 'lastCheck': '7.2', 'code': 0, 'valid': True}}
>Thread-15220::DEBUG::2013-03-18 07:46:42,443::task::1151::TaskManager.Task::(prepare) Task=`96bc048f-364c-4c7b-ab48-df1cfd73c0f3`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0567080974579', 'lastCheck': '8.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '0.3', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0394651889801', 'lastCheck': '7.2', 'code': 0, 'valid': True}}
>Thread-15220::DEBUG::2013-03-18 07:46:42,443::task::568::TaskManager.Task::(_updateState) Task=`96bc048f-364c-4c7b-ab48-df1cfd73c0f3`::moving from state preparing -> state finished
>Thread-15220::DEBUG::2013-03-18 07:46:42,444::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15220::DEBUG::2013-03-18 07:46:42,444::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15220::DEBUG::2013-03-18 07:46:42,444::task::957::TaskManager.Task::(_decref) Task=`96bc048f-364c-4c7b-ab48-df1cfd73c0f3`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:46:44,317::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:46:44,343::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794063 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:46:45,291::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:46:45,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102484 s, 4.0 MB/s\n'; <rc> = 0
>Thread-15225::DEBUG::2013-03-18 07:46:49,532::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15225::DEBUG::2013-03-18 07:46:49,533::task::568::TaskManager.Task::(_updateState) Task=`6022be3a-d8ae-4c51-ae71-97b0b0159db1`::moving from state init -> state preparing
>Thread-15225::INFO::2013-03-18 07:46:49,534::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15225::INFO::2013-03-18 07:46:49,535::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15225::DEBUG::2013-03-18 07:46:49,535::task::1151::TaskManager.Task::(prepare) Task=`6022be3a-d8ae-4c51-ae71-97b0b0159db1`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15225::DEBUG::2013-03-18 07:46:49,535::task::568::TaskManager.Task::(_updateState) Task=`6022be3a-d8ae-4c51-ae71-97b0b0159db1`::moving from state preparing -> state finished
>Thread-15225::DEBUG::2013-03-18 07:46:49,535::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15225::DEBUG::2013-03-18 07:46:49,536::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15225::DEBUG::2013-03-18 07:46:49,536::task::957::TaskManager.Task::(_decref) Task=`6022be3a-d8ae-4c51-ae71-97b0b0159db1`::ref 0 aborting False
>Thread-21::DEBUG::2013-03-18 07:46:52,170::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 07:46:52,171::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:46:52,172::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:46:52,597::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:46:52,604::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:46:52,606::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-15228::DEBUG::2013-03-18 07:46:52,688::task::568::TaskManager.Task::(_updateState) Task=`852761ea-6344-47f4-808d-a816291cdb39`::moving from state init -> state preparing
>Thread-15228::INFO::2013-03-18 07:46:52,690::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15228::INFO::2013-03-18 07:46:52,691::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0290229320526', 'lastCheck': '8.3', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '10.5', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.031406879425', 'lastCheck': '7.4', 'code': 0, 'valid': True}}
>Thread-15228::DEBUG::2013-03-18 07:46:52,691::task::1151::TaskManager.Task::(prepare) Task=`852761ea-6344-47f4-808d-a816291cdb39`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0290229320526', 'lastCheck': '8.3', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '10.5', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.031406879425', 'lastCheck': '7.4', 'code': 0, 'valid': True}}
>Thread-15228::DEBUG::2013-03-18 07:46:52,692::task::568::TaskManager.Task::(_updateState) Task=`852761ea-6344-47f4-808d-a816291cdb39`::moving from state preparing -> state finished
>Thread-15228::DEBUG::2013-03-18 07:46:52,692::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15228::DEBUG::2013-03-18 07:46:52,693::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15228::DEBUG::2013-03-18 07:46:52,693::task::957::TaskManager.Task::(_decref) Task=`852761ea-6344-47f4-808d-a816291cdb39`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:46:54,384::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:46:54,412::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077483 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:46:55,327::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:46:55,365::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00420136 s, 975 kB/s\n'; <rc> = 0
>Thread-15233::DEBUG::2013-03-18 07:46:59,590::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15233::DEBUG::2013-03-18 07:46:59,591::task::568::TaskManager.Task::(_updateState) Task=`9ce19a80-d2bf-4176-b6c7-1f391f4e399f`::moving from state init -> state preparing
>Thread-15233::INFO::2013-03-18 07:46:59,592::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15233::INFO::2013-03-18 07:46:59,593::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15233::DEBUG::2013-03-18 07:46:59,593::task::1151::TaskManager.Task::(prepare) Task=`9ce19a80-d2bf-4176-b6c7-1f391f4e399f`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15233::DEBUG::2013-03-18 07:46:59,594::task::568::TaskManager.Task::(_updateState) Task=`9ce19a80-d2bf-4176-b6c7-1f391f4e399f`::moving from state preparing -> state finished
>Thread-15233::DEBUG::2013-03-18 07:46:59,594::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15233::DEBUG::2013-03-18 07:46:59,595::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15233::DEBUG::2013-03-18 07:46:59,596::task::957::TaskManager.Task::(_decref) Task=`9ce19a80-d2bf-4176-b6c7-1f391f4e399f`::ref 0 aborting False
>Thread-15235::DEBUG::2013-03-18 07:47:02,936::task::568::TaskManager.Task::(_updateState) Task=`b21bcf07-4517-4dd4-9795-1f174e2f1aff`::moving from state init -> state preparing
>Thread-15235::INFO::2013-03-18 07:47:02,936::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15235::INFO::2013-03-18 07:47:02,937::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0301580429077', 'lastCheck': '8.5', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '20.8', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0403292179108', 'lastCheck': '7.6', 'code': 0, 'valid': True}}
>Thread-15235::DEBUG::2013-03-18 07:47:02,937::task::1151::TaskManager.Task::(prepare) Task=`b21bcf07-4517-4dd4-9795-1f174e2f1aff`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0301580429077', 'lastCheck': '8.5', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '20.8', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0403292179108', 'lastCheck': '7.6', 'code': 0, 'valid': True}}
>Thread-15235::DEBUG::2013-03-18 07:47:02,937::task::568::TaskManager.Task::(_updateState) Task=`b21bcf07-4517-4dd4-9795-1f174e2f1aff`::moving from state preparing -> state finished
>Thread-15235::DEBUG::2013-03-18 07:47:02,937::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15235::DEBUG::2013-03-18 07:47:02,937::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15235::DEBUG::2013-03-18 07:47:02,938::task::957::TaskManager.Task::(_decref) Task=`b21bcf07-4517-4dd4-9795-1f174e2f1aff`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:47:04,456::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:47:04,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000839538 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:47:05,370::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:47:05,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0180166 s, 227 kB/s\n'; <rc> = 0
>Thread-15241::DEBUG::2013-03-18 07:47:13,317::task::568::TaskManager.Task::(_updateState) Task=`38e767d6-2392-4bdb-bee6-0aebee4d0601`::moving from state init -> state preparing
>Thread-15241::INFO::2013-03-18 07:47:13,318::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15241::INFO::2013-03-18 07:47:13,319::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.103179931641', 'lastCheck': '8.7', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '31.1', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0502140522003', 'lastCheck': '7.9', 'code': 0, 'valid': True}}
>Thread-15241::DEBUG::2013-03-18 07:47:13,319::task::1151::TaskManager.Task::(prepare) Task=`38e767d6-2392-4bdb-bee6-0aebee4d0601`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.103179931641', 'lastCheck': '8.7', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '31.1', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0502140522003', 'lastCheck': '7.9', 'code': 0, 'valid': True}}
>Thread-15241::DEBUG::2013-03-18 07:47:13,319::task::568::TaskManager.Task::(_updateState) Task=`38e767d6-2392-4bdb-bee6-0aebee4d0601`::moving from state preparing -> state finished
>Thread-15241::DEBUG::2013-03-18 07:47:13,319::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15241::DEBUG::2013-03-18 07:47:13,320::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15241::DEBUG::2013-03-18 07:47:13,320::task::957::TaskManager.Task::(_decref) Task=`38e767d6-2392-4bdb-bee6-0aebee4d0601`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:47:14,598::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:47:14,623::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798496 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:47:15,424::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:47:15,463::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00133325 s, 3.1 MB/s\n'; <rc> = 0
>Thread-15246::DEBUG::2013-03-18 07:47:19,719::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15246::DEBUG::2013-03-18 07:47:19,721::task::568::TaskManager.Task::(_updateState) Task=`5000eb6b-d9f2-4ac8-a857-62189949b300`::moving from state init -> state preparing
>Thread-15246::INFO::2013-03-18 07:47:19,722::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15246::INFO::2013-03-18 07:47:19,723::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15246::DEBUG::2013-03-18 07:47:19,723::task::1151::TaskManager.Task::(prepare) Task=`5000eb6b-d9f2-4ac8-a857-62189949b300`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15246::DEBUG::2013-03-18 07:47:19,723::task::568::TaskManager.Task::(_updateState) Task=`5000eb6b-d9f2-4ac8-a857-62189949b300`::moving from state preparing -> state finished
>Thread-15246::DEBUG::2013-03-18 07:47:19,724::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15246::DEBUG::2013-03-18 07:47:19,724::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15246::DEBUG::2013-03-18 07:47:19,724::task::957::TaskManager.Task::(_decref) Task=`5000eb6b-d9f2-4ac8-a857-62189949b300`::ref 0 aborting False
>Thread-15248::DEBUG::2013-03-18 07:47:23,613::task::568::TaskManager.Task::(_updateState) Task=`a77609b3-cfef-4f3b-b943-c7b7ac84d39a`::moving from state init -> state preparing
>Thread-15248::INFO::2013-03-18 07:47:23,614::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15248::INFO::2013-03-18 07:47:23,615::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0274860858917', 'lastCheck': '9.0', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '41.4', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0406980514526', 'lastCheck': '8.1', 'code': 0, 'valid': True}}
>Thread-15248::DEBUG::2013-03-18 07:47:23,615::task::1151::TaskManager.Task::(prepare) Task=`a77609b3-cfef-4f3b-b943-c7b7ac84d39a`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0274860858917', 'lastCheck': '9.0', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '41.4', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0406980514526', 'lastCheck': '8.1', 'code': 0, 'valid': True}}
>Thread-15248::DEBUG::2013-03-18 07:47:23,615::task::568::TaskManager.Task::(_updateState) Task=`a77609b3-cfef-4f3b-b943-c7b7ac84d39a`::moving from state preparing -> state finished
>Thread-15248::DEBUG::2013-03-18 07:47:23,616::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15248::DEBUG::2013-03-18 07:47:23,616::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15248::DEBUG::2013-03-18 07:47:23,616::task::957::TaskManager.Task::(_decref) Task=`a77609b3-cfef-4f3b-b943-c7b7ac84d39a`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:47:24,664::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:47:24,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764109 s, 5.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:47:25,482::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:47:25,516::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000935744 s, 4.4 MB/s\n'; <rc> = 0
>Thread-15252::DEBUG::2013-03-18 07:47:29,762::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15252::DEBUG::2013-03-18 07:47:29,764::task::568::TaskManager.Task::(_updateState) Task=`902ffcb8-1d81-410d-832c-db2b9b755340`::moving from state init -> state preparing
>Thread-15252::INFO::2013-03-18 07:47:29,765::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15252::INFO::2013-03-18 07:47:29,765::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15252::DEBUG::2013-03-18 07:47:29,766::task::1151::TaskManager.Task::(prepare) Task=`902ffcb8-1d81-410d-832c-db2b9b755340`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15252::DEBUG::2013-03-18 07:47:29,766::task::568::TaskManager.Task::(_updateState) Task=`902ffcb8-1d81-410d-832c-db2b9b755340`::moving from state preparing -> state finished
>Thread-15252::DEBUG::2013-03-18 07:47:29,766::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15252::DEBUG::2013-03-18 07:47:29,767::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15252::DEBUG::2013-03-18 07:47:29,767::task::957::TaskManager.Task::(_decref) Task=`902ffcb8-1d81-410d-832c-db2b9b755340`::ref 0 aborting False
>Thread-15255::DEBUG::2013-03-18 07:47:33,944::task::568::TaskManager.Task::(_updateState) Task=`9dc45853-f4d5-4f43-bdeb-1b9b1fc606f9`::moving from state init -> state preparing
>Thread-15255::INFO::2013-03-18 07:47:33,944::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15255::INFO::2013-03-18 07:47:33,945::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0310320854187', 'lastCheck': '9.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '51.8', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0362050533295', 'lastCheck': '8.4', 'code': 0, 'valid': True}}
>Thread-15255::DEBUG::2013-03-18 07:47:33,945::task::1151::TaskManager.Task::(prepare) Task=`9dc45853-f4d5-4f43-bdeb-1b9b1fc606f9`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0310320854187', 'lastCheck': '9.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '51.8', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0362050533295', 'lastCheck': '8.4', 'code': 0, 'valid': True}}
>Thread-15255::DEBUG::2013-03-18 07:47:33,946::task::568::TaskManager.Task::(_updateState) Task=`9dc45853-f4d5-4f43-bdeb-1b9b1fc606f9`::moving from state preparing -> state finished
>Thread-15255::DEBUG::2013-03-18 07:47:33,946::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15255::DEBUG::2013-03-18 07:47:33,946::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15255::DEBUG::2013-03-18 07:47:33,947::task::957::TaskManager.Task::(_decref) Task=`9dc45853-f4d5-4f43-bdeb-1b9b1fc606f9`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:47:34,733::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:47:34,775::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00160719 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:47:35,522::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:47:35,559::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00287742 s, 1.4 MB/s\n'; <rc> = 0
>Thread-15261::DEBUG::2013-03-18 07:47:44,273::task::568::TaskManager.Task::(_updateState) Task=`63902a40-b846-48f2-b46c-6ff322e1fe56`::moving from state init -> state preparing
>Thread-15261::INFO::2013-03-18 07:47:44,274::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15261::INFO::2013-03-18 07:47:44,275::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0445339679718', 'lastCheck': '9.5', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '62.1', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0382559299469', 'lastCheck': '8.7', 'code': 0, 'valid': True}}
>Thread-15261::DEBUG::2013-03-18 07:47:44,275::task::1151::TaskManager.Task::(prepare) Task=`63902a40-b846-48f2-b46c-6ff322e1fe56`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0445339679718', 'lastCheck': '9.5', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '62.1', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0382559299469', 'lastCheck': '8.7', 'code': 0, 'valid': True}}
>Thread-15261::DEBUG::2013-03-18 07:47:44,275::task::568::TaskManager.Task::(_updateState) Task=`63902a40-b846-48f2-b46c-6ff322e1fe56`::moving from state preparing -> state finished
>Thread-15261::DEBUG::2013-03-18 07:47:44,275::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15261::DEBUG::2013-03-18 07:47:44,276::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15261::DEBUG::2013-03-18 07:47:44,276::task::957::TaskManager.Task::(_decref) Task=`63902a40-b846-48f2-b46c-6ff322e1fe56`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:47:44,815::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:47:44,862::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00156418 s, 2.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:47:45,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:47:45,618::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0177627 s, 231 kB/s\n'; <rc> = 0
>Thread-15265::DEBUG::2013-03-18 07:47:49,875::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15265::DEBUG::2013-03-18 07:47:49,887::task::568::TaskManager.Task::(_updateState) Task=`8ccebb6e-af54-41f7-a805-cff874085fec`::moving from state init -> state preparing
>Thread-15265::INFO::2013-03-18 07:47:49,888::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15265::INFO::2013-03-18 07:47:49,888::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15265::DEBUG::2013-03-18 07:47:49,889::task::1151::TaskManager.Task::(prepare) Task=`8ccebb6e-af54-41f7-a805-cff874085fec`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15265::DEBUG::2013-03-18 07:47:49,889::task::568::TaskManager.Task::(_updateState) Task=`8ccebb6e-af54-41f7-a805-cff874085fec`::moving from state preparing -> state finished
>Thread-15265::DEBUG::2013-03-18 07:47:49,889::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15265::DEBUG::2013-03-18 07:47:49,889::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15265::DEBUG::2013-03-18 07:47:49,890::task::957::TaskManager.Task::(_decref) Task=`8ccebb6e-af54-41f7-a805-cff874085fec`::ref 0 aborting False
>Thread-15227::WARNING::2013-03-18 07:47:52,643::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:47:52,646::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-15268::DEBUG::2013-03-18 07:47:54,574::task::568::TaskManager.Task::(_updateState) Task=`17308062-72e9-4a00-9edc-94eac9bfa7b3`::moving from state init -> state preparing
>Thread-15268::INFO::2013-03-18 07:47:54,575::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15268::INFO::2013-03-18 07:47:54,575::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0491769313812', 'lastCheck': '9.7', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '1.9', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.056116104126', 'lastCheck': '9.0', 'code': 0, 'valid': True}}
>Thread-15268::DEBUG::2013-03-18 07:47:54,575::task::1151::TaskManager.Task::(prepare) Task=`17308062-72e9-4a00-9edc-94eac9bfa7b3`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0491769313812', 'lastCheck': '9.7', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '1.9', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.056116104126', 'lastCheck': '9.0', 'code': 0, 'valid': True}}
>Thread-15268::DEBUG::2013-03-18 07:47:54,576::task::568::TaskManager.Task::(_updateState) Task=`17308062-72e9-4a00-9edc-94eac9bfa7b3`::moving from state preparing -> state finished
>Thread-15268::DEBUG::2013-03-18 07:47:54,576::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15268::DEBUG::2013-03-18 07:47:54,576::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15268::DEBUG::2013-03-18 07:47:54,577::task::957::TaskManager.Task::(_decref) Task=`17308062-72e9-4a00-9edc-94eac9bfa7b3`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:47:54,903::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:47:54,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0245783 s, 167 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:47:55,624::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:47:55,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000929197 s, 4.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:48:02,648::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:48:02,649::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:48:03,105::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:48:03,112::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:48:03,113::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-15275::DEBUG::2013-03-18 07:48:04,901::task::568::TaskManager.Task::(_updateState) Task=`9c4e7d42-c0bc-4987-8253-a2ce2beb9a76`::moving from state init -> state preparing
>Thread-15275::INFO::2013-03-18 07:48:04,902::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15275::INFO::2013-03-18 07:48:04,903::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0596389770508', 'lastCheck': '9.9', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '12.3', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0309309959412', 'lastCheck': '9.2', 'code': 0, 'valid': True}}
>Thread-15275::DEBUG::2013-03-18 07:48:04,903::task::1151::TaskManager.Task::(prepare) Task=`9c4e7d42-c0bc-4987-8253-a2ce2beb9a76`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0596389770508', 'lastCheck': '9.9', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '12.3', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0309309959412', 'lastCheck': '9.2', 'code': 0, 'valid': True}}
>Thread-15275::DEBUG::2013-03-18 07:48:04,904::task::568::TaskManager.Task::(_updateState) Task=`9c4e7d42-c0bc-4987-8253-a2ce2beb9a76`::moving from state preparing -> state finished
>Thread-15275::DEBUG::2013-03-18 07:48:04,904::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15275::DEBUG::2013-03-18 07:48:04,904::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15275::DEBUG::2013-03-18 07:48:04,905::task::957::TaskManager.Task::(_decref) Task=`9c4e7d42-c0bc-4987-8253-a2ce2beb9a76`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:48:05,001::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:48:05,043::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170073 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:48:05,658::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:48:05,700::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.014551 s, 281 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:48:15,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:48:15,119::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000798668 s, 5.1 MB/s\n'; <rc> = 0
>Thread-15281::DEBUG::2013-03-18 07:48:15,293::task::568::TaskManager.Task::(_updateState) Task=`7a535937-5caa-42b6-89a4-5391e15b25dd`::moving from state init -> state preparing
>Thread-15281::INFO::2013-03-18 07:48:15,294::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15281::INFO::2013-03-18 07:48:15,294::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.032623052597', 'lastCheck': '0.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '22.6', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0436220169067', 'lastCheck': '9.6', 'code': 0, 'valid': True}}
>Thread-15281::DEBUG::2013-03-18 07:48:15,295::task::1151::TaskManager.Task::(prepare) Task=`7a535937-5caa-42b6-89a4-5391e15b25dd`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.032623052597', 'lastCheck': '0.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '22.6', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0436220169067', 'lastCheck': '9.6', 'code': 0, 'valid': True}}
>Thread-15281::DEBUG::2013-03-18 07:48:15,295::task::568::TaskManager.Task::(_updateState) Task=`7a535937-5caa-42b6-89a4-5391e15b25dd`::moving from state preparing -> state finished
>Thread-15281::DEBUG::2013-03-18 07:48:15,295::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15281::DEBUG::2013-03-18 07:48:15,295::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15281::DEBUG::2013-03-18 07:48:15,296::task::957::TaskManager.Task::(_decref) Task=`7a535937-5caa-42b6-89a4-5391e15b25dd`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:48:15,705::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:48:15,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794566 s, 5.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:48:25,164::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:48:25,206::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131559 s, 3.1 MB/s\n'; <rc> = 0
>Thread-15287::DEBUG::2013-03-18 07:48:25,622::task::568::TaskManager.Task::(_updateState) Task=`99fdc286-959d-4255-977f-41c78b1307fa`::moving from state init -> state preparing
>Thread-15287::INFO::2013-03-18 07:48:25,623::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15287::INFO::2013-03-18 07:48:25,623::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0435309410095', 'lastCheck': '0.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '33.0', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0314290523529', 'lastCheck': '9.9', 'code': 0, 'valid': True}}
>Thread-15287::DEBUG::2013-03-18 07:48:25,624::task::1151::TaskManager.Task::(prepare) Task=`99fdc286-959d-4255-977f-41c78b1307fa`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0435309410095', 'lastCheck': '0.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '33.0', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0314290523529', 'lastCheck': '9.9', 'code': 0, 'valid': True}}
>Thread-15287::DEBUG::2013-03-18 07:48:25,624::task::568::TaskManager.Task::(_updateState) Task=`99fdc286-959d-4255-977f-41c78b1307fa`::moving from state preparing -> state finished
>Thread-15287::DEBUG::2013-03-18 07:48:25,624::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15287::DEBUG::2013-03-18 07:48:25,624::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15287::DEBUG::2013-03-18 07:48:25,625::task::957::TaskManager.Task::(_decref) Task=`99fdc286-959d-4255-977f-41c78b1307fa`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:48:25,740::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:48:25,788::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188643 s, 217 kB/s\n'; <rc> = 0
>Thread-15291::DEBUG::2013-03-18 07:48:30,146::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15291::DEBUG::2013-03-18 07:48:30,148::task::568::TaskManager.Task::(_updateState) Task=`591b7a18-1a9a-410f-8701-42ddb54a15bb`::moving from state init -> state preparing
>Thread-15291::INFO::2013-03-18 07:48:30,149::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15291::INFO::2013-03-18 07:48:30,150::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15291::DEBUG::2013-03-18 07:48:30,150::task::1151::TaskManager.Task::(prepare) Task=`591b7a18-1a9a-410f-8701-42ddb54a15bb`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15291::DEBUG::2013-03-18 07:48:30,151::task::568::TaskManager.Task::(_updateState) Task=`591b7a18-1a9a-410f-8701-42ddb54a15bb`::moving from state preparing -> state finished
>Thread-15291::DEBUG::2013-03-18 07:48:30,151::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15291::DEBUG::2013-03-18 07:48:30,151::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15291::DEBUG::2013-03-18 07:48:30,151::task::957::TaskManager.Task::(_decref) Task=`591b7a18-1a9a-410f-8701-42ddb54a15bb`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:48:35,246::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:48:35,288::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00145885 s, 2.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:48:35,793::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 07:48:35,795::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 07:48:35,796::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:48:35,830::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00304513 s, 673 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:48:35,832::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00304513 s, 673 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 07:48:35,832::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 07:48:35,832::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 07:48:35,834::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 07:48:35,835::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:48:35,836::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:48:35,836::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:48:35,836::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:48:35,837::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:48:35,838::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-15294::DEBUG::2013-03-18 07:48:35,969::task::568::TaskManager.Task::(_updateState) Task=`750e8b40-9498-40a8-b26e-56620499d447`::moving from state init -> state preparing
>Thread-15294::INFO::2013-03-18 07:48:35,970::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15294::INFO::2013-03-18 07:48:35,971::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0443539619446', 'lastCheck': '0.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '43.3', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0506720542908', 'lastCheck': '10.2', 'code': 0, 'valid': True}}
>Thread-15294::DEBUG::2013-03-18 07:48:35,972::task::1151::TaskManager.Task::(prepare) Task=`750e8b40-9498-40a8-b26e-56620499d447`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0443539619446', 'lastCheck': '0.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '43.3', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0506720542908', 'lastCheck': '10.2', 'code': 0, 'valid': True}}
>Thread-15294::DEBUG::2013-03-18 07:48:35,972::task::568::TaskManager.Task::(_updateState) Task=`750e8b40-9498-40a8-b26e-56620499d447`::moving from state preparing -> state finished
>Thread-15294::DEBUG::2013-03-18 07:48:35,972::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15294::DEBUG::2013-03-18 07:48:35,973::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15294::DEBUG::2013-03-18 07:48:35,973::task::957::TaskManager.Task::(_decref) Task=`750e8b40-9498-40a8-b26e-56620499d447`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:48:36,320::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:48:36,323::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:48:36,324::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::DEBUG::2013-03-18 07:48:36,325::lvm::339::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:48:36,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm pvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,size,vg_name,vg_uuid,pe_start,pe_count,pe_alloc_count,mda_count,dev_size' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:48:36,735::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:48:36,737::lvm::364::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::WARNING::2013-03-18 07:48:36,738::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 07:48:36,739::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 07:48:36,740::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 07:48:36,741::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:48:37,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:48:37,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:48:37,160::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0149976 s, 273 kB/s\n'; <rc> = 0
>Thread-15297::DEBUG::2013-03-18 07:48:40,209::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15297::DEBUG::2013-03-18 07:48:40,214::task::568::TaskManager.Task::(_updateState) Task=`02e5d195-46ed-4a66-acf9-a9bcdc1091ea`::moving from state init -> state preparing
>Thread-15297::INFO::2013-03-18 07:48:40,217::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15297::INFO::2013-03-18 07:48:40,219::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15297::DEBUG::2013-03-18 07:48:40,220::task::1151::TaskManager.Task::(prepare) Task=`02e5d195-46ed-4a66-acf9-a9bcdc1091ea`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15297::DEBUG::2013-03-18 07:48:40,221::task::568::TaskManager.Task::(_updateState) Task=`02e5d195-46ed-4a66-acf9-a9bcdc1091ea`::moving from state preparing -> state finished
>Thread-15297::DEBUG::2013-03-18 07:48:40,221::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15297::DEBUG::2013-03-18 07:48:40,221::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15297::DEBUG::2013-03-18 07:48:40,222::task::957::TaskManager.Task::(_decref) Task=`02e5d195-46ed-4a66-acf9-a9bcdc1091ea`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:48:45,328::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 07:48:45,329::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 07:48:45,330::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:48:45,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00132922 s, 1.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:48:45,374::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00132922 s, 1.5 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 07:48:45,375::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 07:48:45,375::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 07:48:45,377::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 07:48:45,378::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:48:45,378::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:48:45,379::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:48:45,379::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:48:45,380::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:48:45,380::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:48:45,810::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:48:45,815::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:48:45,816::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', '_SHA_CKSUM=dc9056f88d02638478b1a7cc9df8dec53bde4dc6', 'POOL_SPM_ID=3', 'POOL_SPM_LVER=4']
>Thread-20::WARNING::2013-03-18 07:48:45,818::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 07:48:45,820::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 07:48:45,820::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 07:48:45,823::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:48:46,226::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:48:46,228::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:48:46,271::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115636 s, 354 kB/s\n'; <rc> = 0
>Thread-15301::DEBUG::2013-03-18 07:48:46,368::task::568::TaskManager.Task::(_updateState) Task=`68be09ae-95d7-442f-93a1-c8a892304fb4`::moving from state init -> state preparing
>Thread-15301::INFO::2013-03-18 07:48:46,373::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15301::INFO::2013-03-18 07:48:46,378::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0443539619446', 'lastCheck': '11.0', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '53.7', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0504570007324', 'lastCheck': '9.2', 'code': 0, 'valid': True}}
>Thread-15301::DEBUG::2013-03-18 07:48:46,379::task::1151::TaskManager.Task::(prepare) Task=`68be09ae-95d7-442f-93a1-c8a892304fb4`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0443539619446', 'lastCheck': '11.0', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '53.7', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0504570007324', 'lastCheck': '9.2', 'code': 0, 'valid': True}}
>Thread-15301::DEBUG::2013-03-18 07:48:46,383::task::568::TaskManager.Task::(_updateState) Task=`68be09ae-95d7-442f-93a1-c8a892304fb4`::moving from state preparing -> state finished
>Thread-15301::DEBUG::2013-03-18 07:48:46,384::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15301::DEBUG::2013-03-18 07:48:46,386::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15301::DEBUG::2013-03-18 07:48:46,388::task::957::TaskManager.Task::(_decref) Task=`68be09ae-95d7-442f-93a1-c8a892304fb4`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:48:47,165::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:48:47,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0154363 s, 265 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:48:56,410::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:48:56,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000726759 s, 5.6 MB/s\n'; <rc> = 0
>Thread-15307::DEBUG::2013-03-18 07:48:56,653::task::568::TaskManager.Task::(_updateState) Task=`e41e77bf-0608-4f93-b399-5b004e6330f3`::moving from state init -> state preparing
>Thread-15307::INFO::2013-03-18 07:48:56,653::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15307::INFO::2013-03-18 07:48:56,654::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0333790779114', 'lastCheck': '0.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '64.0', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0644950866699', 'lastCheck': '9.4', 'code': 0, 'valid': True}}
>Thread-15307::DEBUG::2013-03-18 07:48:56,654::task::1151::TaskManager.Task::(prepare) Task=`e41e77bf-0608-4f93-b399-5b004e6330f3`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0333790779114', 'lastCheck': '0.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '64.0', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0644950866699', 'lastCheck': '9.4', 'code': 0, 'valid': True}}
>Thread-15307::DEBUG::2013-03-18 07:48:56,655::task::568::TaskManager.Task::(_updateState) Task=`e41e77bf-0608-4f93-b399-5b004e6330f3`::moving from state preparing -> state finished
>Thread-15307::DEBUG::2013-03-18 07:48:56,655::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15307::DEBUG::2013-03-18 07:48:56,655::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15307::DEBUG::2013-03-18 07:48:56,656::task::957::TaskManager.Task::(_decref) Task=`e41e77bf-0608-4f93-b399-5b004e6330f3`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:48:57,233::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:48:57,298::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087509 s, 4.7 MB/s\n'; <rc> = 0
>Thread-15274::WARNING::2013-03-18 07:49:03,139::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:49:03,142::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:49:06,490::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:49:06,533::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154699 s, 2.6 MB/s\n'; <rc> = 0
>Thread-15313::DEBUG::2013-03-18 07:49:06,963::task::568::TaskManager.Task::(_updateState) Task=`70c35c89-f9bc-4523-8c7b-80e975a9517e`::moving from state init -> state preparing
>Thread-15313::INFO::2013-03-18 07:49:06,963::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15313::INFO::2013-03-18 07:49:06,964::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0445740222931', 'lastCheck': '0.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '3.8', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0670759677887', 'lastCheck': '9.7', 'code': 0, 'valid': True}}
>Thread-15313::DEBUG::2013-03-18 07:49:06,964::task::1151::TaskManager.Task::(prepare) Task=`70c35c89-f9bc-4523-8c7b-80e975a9517e`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0445740222931', 'lastCheck': '0.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '3.8', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0670759677887', 'lastCheck': '9.7', 'code': 0, 'valid': True}}
>Thread-15313::DEBUG::2013-03-18 07:49:06,965::task::568::TaskManager.Task::(_updateState) Task=`70c35c89-f9bc-4523-8c7b-80e975a9517e`::moving from state preparing -> state finished
>Thread-15313::DEBUG::2013-03-18 07:49:06,965::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15313::DEBUG::2013-03-18 07:49:06,965::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15313::DEBUG::2013-03-18 07:49:06,966::task::957::TaskManager.Task::(_decref) Task=`70c35c89-f9bc-4523-8c7b-80e975a9517e`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:49:07,306::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:49:07,354::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000838376 s, 4.9 MB/s\n'; <rc> = 0
>Thread-15316::DEBUG::2013-03-18 07:49:10,379::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15316::DEBUG::2013-03-18 07:49:10,381::task::568::TaskManager.Task::(_updateState) Task=`7243e67b-8a9e-4d31-928e-75edb1125abf`::moving from state init -> state preparing
>Thread-15316::INFO::2013-03-18 07:49:10,382::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15316::INFO::2013-03-18 07:49:10,383::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15316::DEBUG::2013-03-18 07:49:10,383::task::1151::TaskManager.Task::(prepare) Task=`7243e67b-8a9e-4d31-928e-75edb1125abf`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15316::DEBUG::2013-03-18 07:49:10,384::task::568::TaskManager.Task::(_updateState) Task=`7243e67b-8a9e-4d31-928e-75edb1125abf`::moving from state preparing -> state finished
>Thread-15316::DEBUG::2013-03-18 07:49:10,384::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15316::DEBUG::2013-03-18 07:49:10,384::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15316::DEBUG::2013-03-18 07:49:10,384::task::957::TaskManager.Task::(_decref) Task=`7243e67b-8a9e-4d31-928e-75edb1125abf`::ref 0 aborting False
>Thread-21::DEBUG::2013-03-18 07:49:13,146::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:49:13,147::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:49:13,589::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:49:13,595::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:49:13,597::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:49:16,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:49:16,600::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000898763 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:49:17,372::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:49:17,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.020164 s, 203 kB/s\n'; <rc> = 0
>Thread-15321::DEBUG::2013-03-18 07:49:17,458::task::568::TaskManager.Task::(_updateState) Task=`40f5f703-4602-4da4-b95b-4b3419963034`::moving from state init -> state preparing
>Thread-15321::INFO::2013-03-18 07:49:17,459::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15321::INFO::2013-03-18 07:49:17,460::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0295541286469', 'lastCheck': '0.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '14.3', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0669250488281', 'lastCheck': '0.0', 'code': 0, 'valid': True}}
>Thread-15321::DEBUG::2013-03-18 07:49:17,460::task::1151::TaskManager.Task::(prepare) Task=`40f5f703-4602-4da4-b95b-4b3419963034`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0295541286469', 'lastCheck': '0.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '14.3', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0669250488281', 'lastCheck': '0.0', 'code': 0, 'valid': True}}
>Thread-15321::DEBUG::2013-03-18 07:49:17,461::task::568::TaskManager.Task::(_updateState) Task=`40f5f703-4602-4da4-b95b-4b3419963034`::moving from state preparing -> state finished
>Thread-15321::DEBUG::2013-03-18 07:49:17,461::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15321::DEBUG::2013-03-18 07:49:17,461::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15321::DEBUG::2013-03-18 07:49:17,462::task::957::TaskManager.Task::(_decref) Task=`40f5f703-4602-4da4-b95b-4b3419963034`::ref 0 aborting False
>Thread-15324::DEBUG::2013-03-18 07:49:20,421::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15324::DEBUG::2013-03-18 07:49:20,421::task::568::TaskManager.Task::(_updateState) Task=`19324b1c-45c7-4f30-8415-91d25a76d2fb`::moving from state init -> state preparing
>Thread-15324::INFO::2013-03-18 07:49:20,422::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15324::INFO::2013-03-18 07:49:20,422::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15324::DEBUG::2013-03-18 07:49:20,422::task::1151::TaskManager.Task::(prepare) Task=`19324b1c-45c7-4f30-8415-91d25a76d2fb`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15324::DEBUG::2013-03-18 07:49:20,422::task::568::TaskManager.Task::(_updateState) Task=`19324b1c-45c7-4f30-8415-91d25a76d2fb`::moving from state preparing -> state finished
>Thread-15324::DEBUG::2013-03-18 07:49:20,423::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15324::DEBUG::2013-03-18 07:49:20,423::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15324::DEBUG::2013-03-18 07:49:20,423::task::957::TaskManager.Task::(_decref) Task=`19324b1c-45c7-4f30-8415-91d25a76d2fb`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:49:26,640::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:49:26,669::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000986049 s, 4.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:49:27,442::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:49:27,486::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140287 s, 2.9 MB/s\n'; <rc> = 0
>Thread-15328::DEBUG::2013-03-18 07:49:27,672::task::568::TaskManager.Task::(_updateState) Task=`23d69606-84a8-4ea9-9907-857a912bf56e`::moving from state init -> state preparing
>Thread-15328::INFO::2013-03-18 07:49:27,673::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15328::INFO::2013-03-18 07:49:27,674::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0318801403046', 'lastCheck': '1.0', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '24.5', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0456531047821', 'lastCheck': '0.2', 'code': 0, 'valid': True}}
>Thread-15328::DEBUG::2013-03-18 07:49:27,674::task::1151::TaskManager.Task::(prepare) Task=`23d69606-84a8-4ea9-9907-857a912bf56e`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0318801403046', 'lastCheck': '1.0', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '24.5', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0456531047821', 'lastCheck': '0.2', 'code': 0, 'valid': True}}
>Thread-15328::DEBUG::2013-03-18 07:49:27,675::task::568::TaskManager.Task::(_updateState) Task=`23d69606-84a8-4ea9-9907-857a912bf56e`::moving from state preparing -> state finished
>Thread-15328::DEBUG::2013-03-18 07:49:27,675::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15328::DEBUG::2013-03-18 07:49:27,675::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15328::DEBUG::2013-03-18 07:49:27,675::task::957::TaskManager.Task::(_decref) Task=`23d69606-84a8-4ea9-9907-857a912bf56e`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:49:36,717::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:49:36,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00077159 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:49:37,492::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:49:37,526::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000806531 s, 5.1 MB/s\n'; <rc> = 0
>Thread-15334::DEBUG::2013-03-18 07:49:37,893::task::568::TaskManager.Task::(_updateState) Task=`9138d501-8617-4260-bffe-e4de0c395afb`::moving from state init -> state preparing
>Thread-15334::INFO::2013-03-18 07:49:37,894::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15334::INFO::2013-03-18 07:49:37,895::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0279378890991', 'lastCheck': '1.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '34.7', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0355999469757', 'lastCheck': '0.4', 'code': 0, 'valid': True}}
>Thread-15334::DEBUG::2013-03-18 07:49:37,895::task::1151::TaskManager.Task::(prepare) Task=`9138d501-8617-4260-bffe-e4de0c395afb`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0279378890991', 'lastCheck': '1.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '34.7', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0355999469757', 'lastCheck': '0.4', 'code': 0, 'valid': True}}
>Thread-15334::DEBUG::2013-03-18 07:49:37,895::task::568::TaskManager.Task::(_updateState) Task=`9138d501-8617-4260-bffe-e4de0c395afb`::moving from state preparing -> state finished
>Thread-15334::DEBUG::2013-03-18 07:49:37,896::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15334::DEBUG::2013-03-18 07:49:37,896::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15334::DEBUG::2013-03-18 07:49:37,897::task::957::TaskManager.Task::(_decref) Task=`9138d501-8617-4260-bffe-e4de0c395afb`::ref 0 aborting False
>Thread-15337::DEBUG::2013-03-18 07:49:40,527::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15337::DEBUG::2013-03-18 07:49:40,528::task::568::TaskManager.Task::(_updateState) Task=`d9089e4e-d390-4ba4-9657-7ca8b92c4e54`::moving from state init -> state preparing
>Thread-15337::INFO::2013-03-18 07:49:40,528::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15337::INFO::2013-03-18 07:49:40,529::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15337::DEBUG::2013-03-18 07:49:40,529::task::1151::TaskManager.Task::(prepare) Task=`d9089e4e-d390-4ba4-9657-7ca8b92c4e54`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15337::DEBUG::2013-03-18 07:49:40,530::task::568::TaskManager.Task::(_updateState) Task=`d9089e4e-d390-4ba4-9657-7ca8b92c4e54`::moving from state preparing -> state finished
>Thread-15337::DEBUG::2013-03-18 07:49:40,530::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15337::DEBUG::2013-03-18 07:49:40,530::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15337::DEBUG::2013-03-18 07:49:40,530::task::957::TaskManager.Task::(_decref) Task=`d9089e4e-d390-4ba4-9657-7ca8b92c4e54`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:49:46,783::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:49:46,811::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000900346 s, 4.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:49:47,532::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:49:47,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00141706 s, 2.9 MB/s\n'; <rc> = 0
>Thread-15341::DEBUG::2013-03-18 07:49:48,261::task::568::TaskManager.Task::(_updateState) Task=`d2f37df2-fce2-4bf7-bd48-8f36d8bbcbda`::moving from state init -> state preparing
>Thread-15341::INFO::2013-03-18 07:49:48,268::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15341::INFO::2013-03-18 07:49:48,271::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0300719738007', 'lastCheck': '1.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '45.1', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0378940105438', 'lastCheck': '0.7', 'code': 0, 'valid': True}}
>Thread-15341::DEBUG::2013-03-18 07:49:48,271::task::1151::TaskManager.Task::(prepare) Task=`d2f37df2-fce2-4bf7-bd48-8f36d8bbcbda`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0300719738007', 'lastCheck': '1.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '45.1', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0378940105438', 'lastCheck': '0.7', 'code': 0, 'valid': True}}
>Thread-15341::DEBUG::2013-03-18 07:49:48,271::task::568::TaskManager.Task::(_updateState) Task=`d2f37df2-fce2-4bf7-bd48-8f36d8bbcbda`::moving from state preparing -> state finished
>Thread-15341::DEBUG::2013-03-18 07:49:48,272::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15341::DEBUG::2013-03-18 07:49:48,272::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15341::DEBUG::2013-03-18 07:49:48,274::task::957::TaskManager.Task::(_decref) Task=`d2f37df2-fce2-4bf7-bd48-8f36d8bbcbda`::ref 0 aborting False
>Thread-15344::DEBUG::2013-03-18 07:49:50,577::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15344::DEBUG::2013-03-18 07:49:50,578::task::568::TaskManager.Task::(_updateState) Task=`8f7dfcbf-80bb-4e37-a0bf-88a97a07cf11`::moving from state init -> state preparing
>Thread-15344::INFO::2013-03-18 07:49:50,579::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15344::INFO::2013-03-18 07:49:50,579::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15344::DEBUG::2013-03-18 07:49:50,580::task::1151::TaskManager.Task::(prepare) Task=`8f7dfcbf-80bb-4e37-a0bf-88a97a07cf11`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15344::DEBUG::2013-03-18 07:49:50,580::task::568::TaskManager.Task::(_updateState) Task=`8f7dfcbf-80bb-4e37-a0bf-88a97a07cf11`::moving from state preparing -> state finished
>Thread-15344::DEBUG::2013-03-18 07:49:50,580::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15344::DEBUG::2013-03-18 07:49:50,580::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15344::DEBUG::2013-03-18 07:49:50,581::task::957::TaskManager.Task::(_decref) Task=`8f7dfcbf-80bb-4e37-a0bf-88a97a07cf11`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:49:56,850::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:49:56,891::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00152252 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:49:57,575::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:49:57,616::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00129147 s, 3.2 MB/s\n'; <rc> = 0
>Thread-15348::DEBUG::2013-03-18 07:49:58,539::task::568::TaskManager.Task::(_updateState) Task=`6c4685da-8e87-4cf5-80ff-dfefb05f5e90`::moving from state init -> state preparing
>Thread-15348::INFO::2013-03-18 07:49:58,543::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15348::INFO::2013-03-18 07:49:58,544::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0422642230988', 'lastCheck': '1.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '55.4', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0431499481201', 'lastCheck': '0.9', 'code': 0, 'valid': True}}
>Thread-15348::DEBUG::2013-03-18 07:49:58,544::task::1151::TaskManager.Task::(prepare) Task=`6c4685da-8e87-4cf5-80ff-dfefb05f5e90`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0422642230988', 'lastCheck': '1.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '55.4', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0431499481201', 'lastCheck': '0.9', 'code': 0, 'valid': True}}
>Thread-15348::DEBUG::2013-03-18 07:49:58,545::task::568::TaskManager.Task::(_updateState) Task=`6c4685da-8e87-4cf5-80ff-dfefb05f5e90`::moving from state preparing -> state finished
>Thread-15348::DEBUG::2013-03-18 07:49:58,545::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15348::DEBUG::2013-03-18 07:49:58,545::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15348::DEBUG::2013-03-18 07:49:58,546::task::957::TaskManager.Task::(_decref) Task=`6c4685da-8e87-4cf5-80ff-dfefb05f5e90`::ref 0 aborting False
>Thread-15350::DEBUG::2013-03-18 07:50:00,621::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15350::DEBUG::2013-03-18 07:50:00,622::task::568::TaskManager.Task::(_updateState) Task=`0e3640e3-01fa-406f-9344-9d1180771bd4`::moving from state init -> state preparing
>Thread-15350::INFO::2013-03-18 07:50:00,623::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15350::INFO::2013-03-18 07:50:00,623::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15350::DEBUG::2013-03-18 07:50:00,624::task::1151::TaskManager.Task::(prepare) Task=`0e3640e3-01fa-406f-9344-9d1180771bd4`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15350::DEBUG::2013-03-18 07:50:00,624::task::568::TaskManager.Task::(_updateState) Task=`0e3640e3-01fa-406f-9344-9d1180771bd4`::moving from state preparing -> state finished
>Thread-15350::DEBUG::2013-03-18 07:50:00,624::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15350::DEBUG::2013-03-18 07:50:00,624::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15350::DEBUG::2013-03-18 07:50:00,625::task::957::TaskManager.Task::(_decref) Task=`0e3640e3-01fa-406f-9344-9d1180771bd4`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:50:06,930::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:50:06,980::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0069763 s, 587 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:50:07,621::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:50:07,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00144927 s, 2.8 MB/s\n'; <rc> = 0
>Thread-15355::DEBUG::2013-03-18 07:50:08,845::task::568::TaskManager.Task::(_updateState) Task=`b89ceac2-694c-43fe-839a-95b0a2a6eac4`::moving from state init -> state preparing
>Thread-15355::INFO::2013-03-18 07:50:08,846::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15355::INFO::2013-03-18 07:50:08,847::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0517339706421', 'lastCheck': '1.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '65.7', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0447559356689', 'lastCheck': '1.2', 'code': 0, 'valid': True}}
>Thread-15355::DEBUG::2013-03-18 07:50:08,847::task::1151::TaskManager.Task::(prepare) Task=`b89ceac2-694c-43fe-839a-95b0a2a6eac4`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0517339706421', 'lastCheck': '1.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '65.7', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0447559356689', 'lastCheck': '1.2', 'code': 0, 'valid': True}}
>Thread-15355::DEBUG::2013-03-18 07:50:08,847::task::568::TaskManager.Task::(_updateState) Task=`b89ceac2-694c-43fe-839a-95b0a2a6eac4`::moving from state preparing -> state finished
>Thread-15355::DEBUG::2013-03-18 07:50:08,847::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15355::DEBUG::2013-03-18 07:50:08,848::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15355::DEBUG::2013-03-18 07:50:08,848::task::957::TaskManager.Task::(_decref) Task=`b89ceac2-694c-43fe-839a-95b0a2a6eac4`::ref 0 aborting False
>Thread-15319::WARNING::2013-03-18 07:50:13,626::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:50:13,628::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:50:17,019::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:50:17,046::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000842651 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:50:17,673::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:50:17,702::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852993 s, 4.8 MB/s\n'; <rc> = 0
>Thread-15361::DEBUG::2013-03-18 07:50:19,873::task::568::TaskManager.Task::(_updateState) Task=`e8574844-c77d-48a1-92a7-e2fd04b55a62`::moving from state init -> state preparing
>Thread-15361::INFO::2013-03-18 07:50:19,874::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15361::INFO::2013-03-18 07:50:19,874::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0281131267548', 'lastCheck': '2.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '6.2', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.030935049057', 'lastCheck': '2.2', 'code': 0, 'valid': True}}
>Thread-15361::DEBUG::2013-03-18 07:50:19,874::task::1151::TaskManager.Task::(prepare) Task=`e8574844-c77d-48a1-92a7-e2fd04b55a62`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0281131267548', 'lastCheck': '2.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '6.2', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.030935049057', 'lastCheck': '2.2', 'code': 0, 'valid': True}}
>Thread-15361::DEBUG::2013-03-18 07:50:19,875::task::568::TaskManager.Task::(_updateState) Task=`e8574844-c77d-48a1-92a7-e2fd04b55a62`::moving from state preparing -> state finished
>Thread-15361::DEBUG::2013-03-18 07:50:19,875::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15361::DEBUG::2013-03-18 07:50:19,875::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15361::DEBUG::2013-03-18 07:50:19,875::task::957::TaskManager.Task::(_decref) Task=`e8574844-c77d-48a1-92a7-e2fd04b55a62`::ref 0 aborting False
>Thread-15363::DEBUG::2013-03-18 07:50:20,861::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15363::DEBUG::2013-03-18 07:50:20,862::task::568::TaskManager.Task::(_updateState) Task=`f2038048-ce48-42e7-9178-2629524a8822`::moving from state init -> state preparing
>Thread-15363::INFO::2013-03-18 07:50:20,862::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15363::INFO::2013-03-18 07:50:20,862::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15363::DEBUG::2013-03-18 07:50:20,863::task::1151::TaskManager.Task::(prepare) Task=`f2038048-ce48-42e7-9178-2629524a8822`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15363::DEBUG::2013-03-18 07:50:20,863::task::568::TaskManager.Task::(_updateState) Task=`f2038048-ce48-42e7-9178-2629524a8822`::moving from state preparing -> state finished
>Thread-15363::DEBUG::2013-03-18 07:50:20,863::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15363::DEBUG::2013-03-18 07:50:20,863::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15363::DEBUG::2013-03-18 07:50:20,863::task::957::TaskManager.Task::(_decref) Task=`f2038048-ce48-42e7-9178-2629524a8822`::ref 0 aborting False
>Thread-21::DEBUG::2013-03-18 07:50:23,630::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:50:23,632::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:50:24,064::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:50:24,070::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:50:24,071::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:50:27,086::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:50:27,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00080028 s, 5.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:50:27,707::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:50:27,738::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00291978 s, 1.4 MB/s\n'; <rc> = 0
>Thread-15369::DEBUG::2013-03-18 07:50:30,144::task::568::TaskManager.Task::(_updateState) Task=`b1cb08a3-cca6-4823-a396-a843cee2d453`::moving from state init -> state preparing
>Thread-15369::INFO::2013-03-18 07:50:30,145::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15369::INFO::2013-03-18 07:50:30,146::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.028294801712', 'lastCheck': '3.0', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '16.5', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.033401966095', 'lastCheck': '2.4', 'code': 0, 'valid': True}}
>Thread-15369::DEBUG::2013-03-18 07:50:30,147::task::1151::TaskManager.Task::(prepare) Task=`b1cb08a3-cca6-4823-a396-a843cee2d453`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.028294801712', 'lastCheck': '3.0', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '16.5', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.033401966095', 'lastCheck': '2.4', 'code': 0, 'valid': True}}
>Thread-15369::DEBUG::2013-03-18 07:50:30,147::task::568::TaskManager.Task::(_updateState) Task=`b1cb08a3-cca6-4823-a396-a843cee2d453`::moving from state preparing -> state finished
>Thread-15369::DEBUG::2013-03-18 07:50:30,147::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15369::DEBUG::2013-03-18 07:50:30,148::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15369::DEBUG::2013-03-18 07:50:30,148::task::957::TaskManager.Task::(_decref) Task=`b1cb08a3-cca6-4823-a396-a843cee2d453`::ref 0 aborting False
>Thread-15371::DEBUG::2013-03-18 07:50:30,905::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15371::DEBUG::2013-03-18 07:50:30,906::task::568::TaskManager.Task::(_updateState) Task=`1238fbc0-82c2-4038-a6f7-fe9f90cc598f`::moving from state init -> state preparing
>Thread-15371::INFO::2013-03-18 07:50:30,907::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15371::INFO::2013-03-18 07:50:30,907::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15371::DEBUG::2013-03-18 07:50:30,907::task::1151::TaskManager.Task::(prepare) Task=`1238fbc0-82c2-4038-a6f7-fe9f90cc598f`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15371::DEBUG::2013-03-18 07:50:30,908::task::568::TaskManager.Task::(_updateState) Task=`1238fbc0-82c2-4038-a6f7-fe9f90cc598f`::moving from state preparing -> state finished
>Thread-15371::DEBUG::2013-03-18 07:50:30,908::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15371::DEBUG::2013-03-18 07:50:30,908::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15371::DEBUG::2013-03-18 07:50:30,908::task::957::TaskManager.Task::(_decref) Task=`1238fbc0-82c2-4038-a6f7-fe9f90cc598f`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:50:37,153::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:50:37,180::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00135049 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:50:37,744::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:50:37,786::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00157791 s, 2.6 MB/s\n'; <rc> = 0
>Thread-15376::DEBUG::2013-03-18 07:50:40,462::task::568::TaskManager.Task::(_updateState) Task=`25a85e86-664a-42b7-b923-7ea9f188005a`::moving from state init -> state preparing
>Thread-15376::INFO::2013-03-18 07:50:40,464::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15376::INFO::2013-03-18 07:50:40,467::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0289959907532', 'lastCheck': '3.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '26.8', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0446581840515', 'lastCheck': '2.7', 'code': 0, 'valid': True}}
>Thread-15376::DEBUG::2013-03-18 07:50:40,467::task::1151::TaskManager.Task::(prepare) Task=`25a85e86-664a-42b7-b923-7ea9f188005a`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0289959907532', 'lastCheck': '3.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '26.8', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0446581840515', 'lastCheck': '2.7', 'code': 0, 'valid': True}}
>Thread-15376::DEBUG::2013-03-18 07:50:40,467::task::568::TaskManager.Task::(_updateState) Task=`25a85e86-664a-42b7-b923-7ea9f188005a`::moving from state preparing -> state finished
>Thread-15376::DEBUG::2013-03-18 07:50:40,468::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15376::DEBUG::2013-03-18 07:50:40,468::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15376::DEBUG::2013-03-18 07:50:40,468::task::957::TaskManager.Task::(_decref) Task=`25a85e86-664a-42b7-b923-7ea9f188005a`::ref 0 aborting False
>Thread-15378::DEBUG::2013-03-18 07:50:40,982::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15378::DEBUG::2013-03-18 07:50:40,983::task::568::TaskManager.Task::(_updateState) Task=`c3c36e21-7f28-4205-b25c-97c0aaaad409`::moving from state init -> state preparing
>Thread-15378::INFO::2013-03-18 07:50:40,983::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15378::INFO::2013-03-18 07:50:40,983::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15378::DEBUG::2013-03-18 07:50:40,984::task::1151::TaskManager.Task::(prepare) Task=`c3c36e21-7f28-4205-b25c-97c0aaaad409`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15378::DEBUG::2013-03-18 07:50:40,984::task::568::TaskManager.Task::(_updateState) Task=`c3c36e21-7f28-4205-b25c-97c0aaaad409`::moving from state preparing -> state finished
>Thread-15378::DEBUG::2013-03-18 07:50:40,984::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15378::DEBUG::2013-03-18 07:50:40,984::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15378::DEBUG::2013-03-18 07:50:40,984::task::957::TaskManager.Task::(_decref) Task=`c3c36e21-7f28-4205-b25c-97c0aaaad409`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:50:47,219::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:50:47,251::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000777761 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:50:47,793::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:50:47,826::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869446 s, 4.7 MB/s\n'; <rc> = 0
>Thread-15383::DEBUG::2013-03-18 07:50:50,774::task::568::TaskManager.Task::(_updateState) Task=`56a083ca-deaa-4980-b6cb-91e0284b488e`::moving from state init -> state preparing
>Thread-15383::INFO::2013-03-18 07:50:50,775::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15383::INFO::2013-03-18 07:50:50,776::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0329949855804', 'lastCheck': '3.5', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '37.1', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0354068279266', 'lastCheck': '2.9', 'code': 0, 'valid': True}}
>Thread-15383::DEBUG::2013-03-18 07:50:50,776::task::1151::TaskManager.Task::(prepare) Task=`56a083ca-deaa-4980-b6cb-91e0284b488e`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0329949855804', 'lastCheck': '3.5', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '37.1', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0354068279266', 'lastCheck': '2.9', 'code': 0, 'valid': True}}
>Thread-15383::DEBUG::2013-03-18 07:50:50,776::task::568::TaskManager.Task::(_updateState) Task=`56a083ca-deaa-4980-b6cb-91e0284b488e`::moving from state preparing -> state finished
>Thread-15383::DEBUG::2013-03-18 07:50:50,777::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15383::DEBUG::2013-03-18 07:50:50,777::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15383::DEBUG::2013-03-18 07:50:50,777::task::957::TaskManager.Task::(_decref) Task=`56a083ca-deaa-4980-b6cb-91e0284b488e`::ref 0 aborting False
>Thread-15385::DEBUG::2013-03-18 07:50:51,030::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15385::DEBUG::2013-03-18 07:50:51,031::task::568::TaskManager.Task::(_updateState) Task=`4ca3e890-fffc-4478-9630-e74966f1bfbb`::moving from state init -> state preparing
>Thread-15385::INFO::2013-03-18 07:50:51,032::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15385::INFO::2013-03-18 07:50:51,032::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15385::DEBUG::2013-03-18 07:50:51,033::task::1151::TaskManager.Task::(prepare) Task=`4ca3e890-fffc-4478-9630-e74966f1bfbb`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15385::DEBUG::2013-03-18 07:50:51,033::task::568::TaskManager.Task::(_updateState) Task=`4ca3e890-fffc-4478-9630-e74966f1bfbb`::moving from state preparing -> state finished
>Thread-15385::DEBUG::2013-03-18 07:50:51,033::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15385::DEBUG::2013-03-18 07:50:51,033::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15385::DEBUG::2013-03-18 07:50:51,034::task::957::TaskManager.Task::(_decref) Task=`4ca3e890-fffc-4478-9630-e74966f1bfbb`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:50:57,297::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:50:57,323::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000872341 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:50:57,832::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:50:57,870::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00222566 s, 1.8 MB/s\n'; <rc> = 0
>Thread-15390::DEBUG::2013-03-18 07:51:01,081::task::568::TaskManager.Task::(_updateState) Task=`42798fc6-810b-45d9-a22c-631aea7f553f`::moving from state init -> state preparing
>Thread-15390::INFO::2013-03-18 07:51:01,082::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15390::INFO::2013-03-18 07:51:01,083::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0285351276398', 'lastCheck': '3.7', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '47.5', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0398490428925', 'lastCheck': '3.2', 'code': 0, 'valid': True}}
>Thread-15390::DEBUG::2013-03-18 07:51:01,083::task::1151::TaskManager.Task::(prepare) Task=`42798fc6-810b-45d9-a22c-631aea7f553f`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0285351276398', 'lastCheck': '3.7', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '47.5', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0398490428925', 'lastCheck': '3.2', 'code': 0, 'valid': True}}
>Thread-15390::DEBUG::2013-03-18 07:51:01,084::task::568::TaskManager.Task::(_updateState) Task=`42798fc6-810b-45d9-a22c-631aea7f553f`::moving from state preparing -> state finished
>Thread-15390::DEBUG::2013-03-18 07:51:01,084::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15390::DEBUG::2013-03-18 07:51:01,084::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15390::DEBUG::2013-03-18 07:51:01,084::task::957::TaskManager.Task::(_decref) Task=`42798fc6-810b-45d9-a22c-631aea7f553f`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:51:07,365::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:51:07,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000765743 s, 5.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:51:07,875::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:51:07,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00146137 s, 2.8 MB/s\n'; <rc> = 0
>Thread-15396::DEBUG::2013-03-18 07:51:11,368::task::568::TaskManager.Task::(_updateState) Task=`9cdcfb0b-9d3b-42a5-8fcb-d30ab6f11d9f`::moving from state init -> state preparing
>Thread-15396::INFO::2013-03-18 07:51:11,369::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15396::INFO::2013-03-18 07:51:11,370::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0302770137787', 'lastCheck': '3.9', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '57.7', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0479950904846', 'lastCheck': '3.4', 'code': 0, 'valid': True}}
>Thread-15396::DEBUG::2013-03-18 07:51:11,370::task::1151::TaskManager.Task::(prepare) Task=`9cdcfb0b-9d3b-42a5-8fcb-d30ab6f11d9f`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0302770137787', 'lastCheck': '3.9', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '57.7', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0479950904846', 'lastCheck': '3.4', 'code': 0, 'valid': True}}
>Thread-15396::DEBUG::2013-03-18 07:51:11,371::task::568::TaskManager.Task::(_updateState) Task=`9cdcfb0b-9d3b-42a5-8fcb-d30ab6f11d9f`::moving from state preparing -> state finished
>Thread-15396::DEBUG::2013-03-18 07:51:11,371::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15396::DEBUG::2013-03-18 07:51:11,371::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15396::DEBUG::2013-03-18 07:51:11,371::task::957::TaskManager.Task::(_decref) Task=`9cdcfb0b-9d3b-42a5-8fcb-d30ab6f11d9f`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:51:17,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:51:17,469::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862808 s, 4.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:51:17,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:51:17,967::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00200705 s, 2.0 MB/s\n'; <rc> = 0
>Thread-15402::DEBUG::2013-03-18 07:51:21,278::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15402::DEBUG::2013-03-18 07:51:21,280::task::568::TaskManager.Task::(_updateState) Task=`3b7befe5-e539-428b-b35c-9d7788962c8e`::moving from state init -> state preparing
>Thread-15402::INFO::2013-03-18 07:51:21,281::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15402::INFO::2013-03-18 07:51:21,281::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15402::DEBUG::2013-03-18 07:51:21,281::task::1151::TaskManager.Task::(prepare) Task=`3b7befe5-e539-428b-b35c-9d7788962c8e`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15402::DEBUG::2013-03-18 07:51:21,282::task::568::TaskManager.Task::(_updateState) Task=`3b7befe5-e539-428b-b35c-9d7788962c8e`::moving from state preparing -> state finished
>Thread-15402::DEBUG::2013-03-18 07:51:21,282::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15402::DEBUG::2013-03-18 07:51:21,282::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15402::DEBUG::2013-03-18 07:51:21,282::task::957::TaskManager.Task::(_decref) Task=`3b7befe5-e539-428b-b35c-9d7788962c8e`::ref 0 aborting False
>Thread-15403::DEBUG::2013-03-18 07:51:21,699::task::568::TaskManager.Task::(_updateState) Task=`85e2144c-ab99-4dc0-85fb-1c2d2dcb9f2e`::moving from state init -> state preparing
>Thread-15403::INFO::2013-03-18 07:51:21,699::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15403::INFO::2013-03-18 07:51:21,700::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.038537979126', 'lastCheck': '4.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '68.1', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0430679321289', 'lastCheck': '3.7', 'code': 0, 'valid': True}}
>Thread-15403::DEBUG::2013-03-18 07:51:21,700::task::1151::TaskManager.Task::(prepare) Task=`85e2144c-ab99-4dc0-85fb-1c2d2dcb9f2e`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.038537979126', 'lastCheck': '4.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '68.1', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0430679321289', 'lastCheck': '3.7', 'code': 0, 'valid': True}}
>Thread-15403::DEBUG::2013-03-18 07:51:21,700::task::568::TaskManager.Task::(_updateState) Task=`85e2144c-ab99-4dc0-85fb-1c2d2dcb9f2e`::moving from state preparing -> state finished
>Thread-15403::DEBUG::2013-03-18 07:51:21,701::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15403::DEBUG::2013-03-18 07:51:21,701::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15403::DEBUG::2013-03-18 07:51:21,701::task::957::TaskManager.Task::(_decref) Task=`85e2144c-ab99-4dc0-85fb-1c2d2dcb9f2e`::ref 0 aborting False
>Thread-15366::WARNING::2013-03-18 07:51:24,097::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:51:24,110::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:51:27,517::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:51:27,565::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170914 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:51:27,977::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:51:28,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000928815 s, 4.4 MB/s\n'; <rc> = 0
>Thread-15409::DEBUG::2013-03-18 07:51:32,004::task::568::TaskManager.Task::(_updateState) Task=`23b05f4a-0811-4c23-b57b-3e917ecd84a4`::moving from state init -> state preparing
>Thread-15409::INFO::2013-03-18 07:51:32,005::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15409::INFO::2013-03-18 07:51:32,006::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0490200519562', 'lastCheck': '4.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '7.9', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0296061038971', 'lastCheck': '4.0', 'code': 0, 'valid': True}}
>Thread-15409::DEBUG::2013-03-18 07:51:32,006::task::1151::TaskManager.Task::(prepare) Task=`23b05f4a-0811-4c23-b57b-3e917ecd84a4`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0490200519562', 'lastCheck': '4.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '7.9', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0296061038971', 'lastCheck': '4.0', 'code': 0, 'valid': True}}
>Thread-15409::DEBUG::2013-03-18 07:51:32,006::task::568::TaskManager.Task::(_updateState) Task=`23b05f4a-0811-4c23-b57b-3e917ecd84a4`::moving from state preparing -> state finished
>Thread-15409::DEBUG::2013-03-18 07:51:32,007::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15409::DEBUG::2013-03-18 07:51:32,007::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15409::DEBUG::2013-03-18 07:51:32,007::task::957::TaskManager.Task::(_decref) Task=`23b05f4a-0811-4c23-b57b-3e917ecd84a4`::ref 0 aborting False
>Thread-21::DEBUG::2013-03-18 07:51:34,113::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:51:34,114::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:51:34,546::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:51:34,552::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:51:34,553::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:51:37,605::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:51:37,652::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00170708 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:51:38,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:51:38,050::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00381152 s, 1.1 MB/s\n'; <rc> = 0
>Thread-15416::DEBUG::2013-03-18 07:51:42,348::task::568::TaskManager.Task::(_updateState) Task=`08cbc3cb-8e61-409c-9411-5c76fa984245`::moving from state init -> state preparing
>Thread-15416::INFO::2013-03-18 07:51:42,349::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15416::INFO::2013-03-18 07:51:42,349::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0488150119781', 'lastCheck': '4.7', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '18.2', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0419631004333', 'lastCheck': '4.3', 'code': 0, 'valid': True}}
>Thread-15416::DEBUG::2013-03-18 07:51:42,350::task::1151::TaskManager.Task::(prepare) Task=`08cbc3cb-8e61-409c-9411-5c76fa984245`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0488150119781', 'lastCheck': '4.7', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '18.2', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0419631004333', 'lastCheck': '4.3', 'code': 0, 'valid': True}}
>Thread-15416::DEBUG::2013-03-18 07:51:42,350::task::568::TaskManager.Task::(_updateState) Task=`08cbc3cb-8e61-409c-9411-5c76fa984245`::moving from state preparing -> state finished
>Thread-15416::DEBUG::2013-03-18 07:51:42,350::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15416::DEBUG::2013-03-18 07:51:42,351::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15416::DEBUG::2013-03-18 07:51:42,351::task::957::TaskManager.Task::(_decref) Task=`08cbc3cb-8e61-409c-9411-5c76fa984245`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:51:47,692::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:51:47,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894683 s, 4.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:51:48,055::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:51:48,084::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00084076 s, 4.9 MB/s\n'; <rc> = 0
>Thread-15422::DEBUG::2013-03-18 07:51:51,411::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15422::DEBUG::2013-03-18 07:51:51,413::task::568::TaskManager.Task::(_updateState) Task=`51d6d711-9d39-46e6-b086-f4acc367a65a`::moving from state init -> state preparing
>Thread-15422::INFO::2013-03-18 07:51:51,414::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15422::INFO::2013-03-18 07:51:51,415::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15422::DEBUG::2013-03-18 07:51:51,415::task::1151::TaskManager.Task::(prepare) Task=`51d6d711-9d39-46e6-b086-f4acc367a65a`::finished: {'spm_st': {'spmId': 3, 'spmStatus': 'Free', 'spmLver': 4}}
>Thread-15422::DEBUG::2013-03-18 07:51:51,415::task::568::TaskManager.Task::(_updateState) Task=`51d6d711-9d39-46e6-b086-f4acc367a65a`::moving from state preparing -> state finished
>Thread-15422::DEBUG::2013-03-18 07:51:51,416::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15422::DEBUG::2013-03-18 07:51:51,416::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15422::DEBUG::2013-03-18 07:51:51,416::task::957::TaskManager.Task::(_decref) Task=`51d6d711-9d39-46e6-b086-f4acc367a65a`::ref 0 aborting False
>Thread-15423::DEBUG::2013-03-18 07:51:52,652::task::568::TaskManager.Task::(_updateState) Task=`3657b4aa-e2cf-4ad0-bb31-20558c9e4aa9`::moving from state init -> state preparing
>Thread-15423::INFO::2013-03-18 07:51:52,652::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15423::INFO::2013-03-18 07:51:52,653::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0378429889679', 'lastCheck': '4.9', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '28.5', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0302078723907', 'lastCheck': '4.6', 'code': 0, 'valid': True}}
>Thread-15423::DEBUG::2013-03-18 07:51:52,653::task::1151::TaskManager.Task::(prepare) Task=`3657b4aa-e2cf-4ad0-bb31-20558c9e4aa9`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0378429889679', 'lastCheck': '4.9', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '28.5', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0302078723907', 'lastCheck': '4.6', 'code': 0, 'valid': True}}
>Thread-15423::DEBUG::2013-03-18 07:51:52,654::task::568::TaskManager.Task::(_updateState) Task=`3657b4aa-e2cf-4ad0-bb31-20558c9e4aa9`::moving from state preparing -> state finished
>Thread-15423::DEBUG::2013-03-18 07:51:52,654::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15423::DEBUG::2013-03-18 07:51:52,654::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15423::DEBUG::2013-03-18 07:51:52,654::task::957::TaskManager.Task::(_decref) Task=`3657b4aa-e2cf-4ad0-bb31-20558c9e4aa9`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:51:57,768::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:51:57,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159883 s, 256 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:51:58,090::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:51:58,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000916729 s, 4.5 MB/s\n'; <rc> = 0
>Thread-15429::DEBUG::2013-03-18 07:52:02,923::task::568::TaskManager.Task::(_updateState) Task=`b30b48d3-b5e2-4883-ba19-d32073d6cc7b`::moving from state init -> state preparing
>Thread-15429::INFO::2013-03-18 07:52:02,931::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15429::INFO::2013-03-18 07:52:02,932::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0496289730072', 'lastCheck': '5.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '38.8', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0305371284485', 'lastCheck': '4.8', 'code': 0, 'valid': True}}
>Thread-15429::DEBUG::2013-03-18 07:52:02,941::task::1151::TaskManager.Task::(prepare) Task=`b30b48d3-b5e2-4883-ba19-d32073d6cc7b`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0496289730072', 'lastCheck': '5.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '38.8', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0305371284485', 'lastCheck': '4.8', 'code': 0, 'valid': True}}
>Thread-15429::DEBUG::2013-03-18 07:52:02,941::task::568::TaskManager.Task::(_updateState) Task=`b30b48d3-b5e2-4883-ba19-d32073d6cc7b`::moving from state preparing -> state finished
>Thread-15429::DEBUG::2013-03-18 07:52:02,941::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15429::DEBUG::2013-03-18 07:52:02,942::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15429::DEBUG::2013-03-18 07:52:02,942::task::957::TaskManager.Task::(_decref) Task=`b30b48d3-b5e2-4883-ba19-d32073d6cc7b`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:52:07,875::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:52:07,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000850712 s, 4.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:52:08,127::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:52:08,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0102291 s, 400 kB/s\n'; <rc> = 0
>Thread-15435::DEBUG::2013-03-18 07:52:13,309::task::568::TaskManager.Task::(_updateState) Task=`59bae24a-f73e-448a-9aa1-991bbd24630f`::moving from state init -> state preparing
>Thread-15435::INFO::2013-03-18 07:52:13,310::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15435::INFO::2013-03-18 07:52:13,311::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0367980003357', 'lastCheck': '5.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '49.2', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0516018867493', 'lastCheck': '5.1', 'code': 0, 'valid': True}}
>Thread-15435::DEBUG::2013-03-18 07:52:13,311::task::1151::TaskManager.Task::(prepare) Task=`59bae24a-f73e-448a-9aa1-991bbd24630f`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0367980003357', 'lastCheck': '5.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '49.2', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0516018867493', 'lastCheck': '5.1', 'code': 0, 'valid': True}}
>Thread-15435::DEBUG::2013-03-18 07:52:13,317::task::568::TaskManager.Task::(_updateState) Task=`59bae24a-f73e-448a-9aa1-991bbd24630f`::moving from state preparing -> state finished
>Thread-15435::DEBUG::2013-03-18 07:52:13,317::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15435::DEBUG::2013-03-18 07:52:13,317::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15435::DEBUG::2013-03-18 07:52:13,318::task::957::TaskManager.Task::(_decref) Task=`59bae24a-f73e-448a-9aa1-991bbd24630f`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:52:17,950::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:52:17,990::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00168363 s, 2.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:52:18,182::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:52:18,227::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000763575 s, 5.4 MB/s\n'; <rc> = 0
>Thread-15441::DEBUG::2013-03-18 07:52:23,817::task::568::TaskManager.Task::(_updateState) Task=`cee76fa8-ec6f-4946-83c9-c73ff1e7cdf2`::moving from state init -> state preparing
>Thread-15441::INFO::2013-03-18 07:52:23,818::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15441::INFO::2013-03-18 07:52:23,818::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0417230129242', 'lastCheck': '5.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '59.7', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0465478897095', 'lastCheck': '5.6', 'code': 0, 'valid': True}}
>Thread-15441::DEBUG::2013-03-18 07:52:23,819::task::1151::TaskManager.Task::(prepare) Task=`cee76fa8-ec6f-4946-83c9-c73ff1e7cdf2`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0417230129242', 'lastCheck': '5.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '59.7', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0465478897095', 'lastCheck': '5.6', 'code': 0, 'valid': True}}
>Thread-15441::DEBUG::2013-03-18 07:52:23,819::task::568::TaskManager.Task::(_updateState) Task=`cee76fa8-ec6f-4946-83c9-c73ff1e7cdf2`::moving from state preparing -> state finished
>Thread-15441::DEBUG::2013-03-18 07:52:23,819::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15441::DEBUG::2013-03-18 07:52:23,820::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15441::DEBUG::2013-03-18 07:52:23,820::task::957::TaskManager.Task::(_decref) Task=`cee76fa8-ec6f-4946-83c9-c73ff1e7cdf2`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:52:28,032::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:52:28,090::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188402 s, 217 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:52:28,232::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:52:28,282::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129505 s, 316 kB/s\n'; <rc> = 0
>Thread-15447::DEBUG::2013-03-18 07:52:34,280::task::568::TaskManager.Task::(_updateState) Task=`afc30918-8d20-4254-9ae7-b984d296e1bb`::moving from state init -> state preparing
>Thread-15447::INFO::2013-03-18 07:52:34,281::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15447::INFO::2013-03-18 07:52:34,282::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0604441165924', 'lastCheck': '6.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '70.2', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0520071983337', 'lastCheck': '6.0', 'code': 0, 'valid': True}}
>Thread-15447::DEBUG::2013-03-18 07:52:34,282::task::1151::TaskManager.Task::(prepare) Task=`afc30918-8d20-4254-9ae7-b984d296e1bb`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0604441165924', 'lastCheck': '6.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '70.2', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0520071983337', 'lastCheck': '6.0', 'code': 0, 'valid': True}}
>Thread-15447::DEBUG::2013-03-18 07:52:34,282::task::568::TaskManager.Task::(_updateState) Task=`afc30918-8d20-4254-9ae7-b984d296e1bb`::moving from state preparing -> state finished
>Thread-15447::DEBUG::2013-03-18 07:52:34,282::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15447::DEBUG::2013-03-18 07:52:34,283::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15447::DEBUG::2013-03-18 07:52:34,283::task::957::TaskManager.Task::(_decref) Task=`afc30918-8d20-4254-9ae7-b984d296e1bb`::ref 0 aborting False
>Thread-15412::WARNING::2013-03-18 07:52:34,579::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:52:34,582::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-20::DEBUG::2013-03-18 07:52:38,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:52:38,246::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00570276 s, 718 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:52:38,402::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:52:38,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000794489 s, 5.2 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:52:44,584::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 07:52:44,585::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:52:44,586::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-15453::DEBUG::2013-03-18 07:52:44,720::task::568::TaskManager.Task::(_updateState) Task=`b0de3776-cdb9-4ba4-88a9-5f51a4cd2acc`::moving from state init -> state preparing
>Thread-15453::INFO::2013-03-18 07:52:44,721::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15453::INFO::2013-03-18 07:52:44,722::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0574998855591', 'lastCheck': '6.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '10.1', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0664300918579', 'lastCheck': '6.2', 'code': 0, 'valid': True}}
>Thread-15453::DEBUG::2013-03-18 07:52:44,723::task::1151::TaskManager.Task::(prepare) Task=`b0de3776-cdb9-4ba4-88a9-5f51a4cd2acc`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0574998855591', 'lastCheck': '6.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '10.1', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0664300918579', 'lastCheck': '6.2', 'code': 0, 'valid': True}}
>Thread-15453::DEBUG::2013-03-18 07:52:44,723::task::568::TaskManager.Task::(_updateState) Task=`b0de3776-cdb9-4ba4-88a9-5f51a4cd2acc`::moving from state preparing -> state finished
>Thread-15453::DEBUG::2013-03-18 07:52:44,723::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15453::DEBUG::2013-03-18 07:52:44,723::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15453::DEBUG::2013-03-18 07:52:44,724::task::957::TaskManager.Task::(_decref) Task=`b0de3776-cdb9-4ba4-88a9-5f51a4cd2acc`::ref 0 aborting False
>Thread-21::DEBUG::2013-03-18 07:52:44,988::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:52:44,994::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:52:44,995::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:52:48,394::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:52:48,436::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146754 s, 279 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:52:48,496::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:52:48,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000799212 s, 5.1 MB/s\n'; <rc> = 0
>Thread-15460::DEBUG::2013-03-18 07:52:54,964::task::568::TaskManager.Task::(_updateState) Task=`fc60dd4b-9e58-4025-9fd1-423b7d51a62d`::moving from state init -> state preparing
>Thread-15460::INFO::2013-03-18 07:52:54,965::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15460::INFO::2013-03-18 07:52:54,966::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0459141731262', 'lastCheck': '6.5', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '20.4', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0359938144684', 'lastCheck': '6.4', 'code': 0, 'valid': True}}
>Thread-15460::DEBUG::2013-03-18 07:52:54,966::task::1151::TaskManager.Task::(prepare) Task=`fc60dd4b-9e58-4025-9fd1-423b7d51a62d`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0459141731262', 'lastCheck': '6.5', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '20.4', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0359938144684', 'lastCheck': '6.4', 'code': 0, 'valid': True}}
>Thread-15460::DEBUG::2013-03-18 07:52:54,967::task::568::TaskManager.Task::(_updateState) Task=`fc60dd4b-9e58-4025-9fd1-423b7d51a62d`::moving from state preparing -> state finished
>Thread-15460::DEBUG::2013-03-18 07:52:54,967::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15460::DEBUG::2013-03-18 07:52:54,967::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15460::DEBUG::2013-03-18 07:52:54,968::task::957::TaskManager.Task::(_decref) Task=`fc60dd4b-9e58-4025-9fd1-423b7d51a62d`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:52:58,479::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:52:58,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150398 s, 2.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:52:58,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:52:58,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0155352 s, 264 kB/s\n'; <rc> = 0
>Thread-15466::DEBUG::2013-03-18 07:53:05,202::task::568::TaskManager.Task::(_updateState) Task=`05334797-10c2-40b9-b42c-945d2cf5c0d0`::moving from state init -> state preparing
>Thread-15466::INFO::2013-03-18 07:53:05,203::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15466::INFO::2013-03-18 07:53:05,204::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0432391166687', 'lastCheck': '6.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '30.6', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0620188713074', 'lastCheck': '6.6', 'code': 0, 'valid': True}}
>Thread-15466::DEBUG::2013-03-18 07:53:05,204::task::1151::TaskManager.Task::(prepare) Task=`05334797-10c2-40b9-b42c-945d2cf5c0d0`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0432391166687', 'lastCheck': '6.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '30.6', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0620188713074', 'lastCheck': '6.6', 'code': 0, 'valid': True}}
>Thread-15466::DEBUG::2013-03-18 07:53:05,204::task::568::TaskManager.Task::(_updateState) Task=`05334797-10c2-40b9-b42c-945d2cf5c0d0`::moving from state preparing -> state finished
>Thread-15466::DEBUG::2013-03-18 07:53:05,206::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15466::DEBUG::2013-03-18 07:53:05,207::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15466::DEBUG::2013-03-18 07:53:05,207::task::957::TaskManager.Task::(_decref) Task=`05334797-10c2-40b9-b42c-945d2cf5c0d0`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:53:08,632::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:53:08,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:53:08,714::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013632 s, 3.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:53:08,737::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00111267 s, 3.7 MB/s\n'; <rc> = 0
>Thread-15472::DEBUG::2013-03-18 07:53:15,510::task::568::TaskManager.Task::(_updateState) Task=`403a2b06-79c4-46a2-86c2-ebfa796c4707`::moving from state init -> state preparing
>Thread-15472::INFO::2013-03-18 07:53:15,511::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15472::INFO::2013-03-18 07:53:15,511::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0628199577332', 'lastCheck': '6.7', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '40.9', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.108231067657', 'lastCheck': '6.8', 'code': 0, 'valid': True}}
>Thread-15472::DEBUG::2013-03-18 07:53:15,512::task::1151::TaskManager.Task::(prepare) Task=`403a2b06-79c4-46a2-86c2-ebfa796c4707`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0628199577332', 'lastCheck': '6.7', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '40.9', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.108231067657', 'lastCheck': '6.8', 'code': 0, 'valid': True}}
>Thread-15472::DEBUG::2013-03-18 07:53:15,512::task::568::TaskManager.Task::(_updateState) Task=`403a2b06-79c4-46a2-86c2-ebfa796c4707`::moving from state preparing -> state finished
>Thread-15472::DEBUG::2013-03-18 07:53:15,512::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15472::DEBUG::2013-03-18 07:53:15,513::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15472::DEBUG::2013-03-18 07:53:15,513::task::957::TaskManager.Task::(_decref) Task=`403a2b06-79c4-46a2-86c2-ebfa796c4707`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:53:18,746::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:53:18,780::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:53:18,838::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000829967 s, 4.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:53:18,859::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00125508 s, 3.3 MB/s\n'; <rc> = 0
>Thread-15478::DEBUG::2013-03-18 07:53:25,774::task::568::TaskManager.Task::(_updateState) Task=`9038fad1-cfde-427c-97eb-451ec8798851`::moving from state init -> state preparing
>Thread-15478::INFO::2013-03-18 07:53:25,775::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15478::INFO::2013-03-18 07:53:25,776::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0661470890045', 'lastCheck': '6.9', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '51.2', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.115926027298', 'lastCheck': '6.9', 'code': 0, 'valid': True}}
>Thread-15478::DEBUG::2013-03-18 07:53:25,776::task::1151::TaskManager.Task::(prepare) Task=`9038fad1-cfde-427c-97eb-451ec8798851`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0661470890045', 'lastCheck': '6.9', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '51.2', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.115926027298', 'lastCheck': '6.9', 'code': 0, 'valid': True}}
>Thread-15478::DEBUG::2013-03-18 07:53:25,776::task::568::TaskManager.Task::(_updateState) Task=`9038fad1-cfde-427c-97eb-451ec8798851`::moving from state preparing -> state finished
>Thread-15478::DEBUG::2013-03-18 07:53:25,777::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15478::DEBUG::2013-03-18 07:53:25,777::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15478::DEBUG::2013-03-18 07:53:25,778::task::957::TaskManager.Task::(_decref) Task=`9038fad1-cfde-427c-97eb-451ec8798851`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:53:28,867::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:53:28,897::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000743368 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:53:28,901::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:53:28,963::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0142415 s, 288 kB/s\n'; <rc> = 0
>Thread-15484::DEBUG::2013-03-18 07:53:36,044::task::568::TaskManager.Task::(_updateState) Task=`219e3536-5932-4137-a415-7a5041d01b87`::moving from state init -> state preparing
>Thread-15484::INFO::2013-03-18 07:53:36,045::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15484::INFO::2013-03-18 07:53:36,045::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0632219314575', 'lastCheck': '7.0', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '61.5', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0316989421844', 'lastCheck': '7.1', 'code': 0, 'valid': True}}
>Thread-15484::DEBUG::2013-03-18 07:53:36,045::task::1151::TaskManager.Task::(prepare) Task=`219e3536-5932-4137-a415-7a5041d01b87`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0632219314575', 'lastCheck': '7.0', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '61.5', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0316989421844', 'lastCheck': '7.1', 'code': 0, 'valid': True}}
>Thread-15484::DEBUG::2013-03-18 07:53:36,046::task::568::TaskManager.Task::(_updateState) Task=`219e3536-5932-4137-a415-7a5041d01b87`::moving from state preparing -> state finished
>Thread-15484::DEBUG::2013-03-18 07:53:36,046::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15484::DEBUG::2013-03-18 07:53:36,046::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15484::DEBUG::2013-03-18 07:53:36,046::task::957::TaskManager.Task::(_decref) Task=`219e3536-5932-4137-a415-7a5041d01b87`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:53:38,925::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 07:53:38,926::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 07:53:38,927::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:53:38,960::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00133017 s, 1.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:53:38,961::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00133017 s, 1.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 07:53:38,962::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 07:53:38,962::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 07:53:38,966::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 07:53:38,967::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:53:38,968::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:53:38,968::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:53:38,969::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:53:38,969::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:53:38,970::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:53:39,044::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:53:39,124::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0193913 s, 211 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:53:39,474::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:53:39,478::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:53:39,479::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 07:53:39,483::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 07:53:39,484::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 07:53:39,484::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 07:53:39,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:53:39,915::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:53:39,918::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:53:39,966::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0127896 s, 320 kB/s\n'; <rc> = 0
>Thread-15455::WARNING::2013-03-18 07:53:45,023::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:53:45,027::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-15490::DEBUG::2013-03-18 07:53:46,317::task::568::TaskManager.Task::(_updateState) Task=`18892297-8024-4fae-9e6b-20128f8150d6`::moving from state init -> state preparing
>Thread-15490::INFO::2013-03-18 07:53:46,318::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15490::INFO::2013-03-18 07:53:46,318::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0827469825745', 'lastCheck': '7.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '1.3', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.050595998764', 'lastCheck': '6.3', 'code': 0, 'valid': True}}
>Thread-15490::DEBUG::2013-03-18 07:53:46,318::task::1151::TaskManager.Task::(prepare) Task=`18892297-8024-4fae-9e6b-20128f8150d6`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0827469825745', 'lastCheck': '7.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '1.3', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.050595998764', 'lastCheck': '6.3', 'code': 0, 'valid': True}}
>Thread-15490::DEBUG::2013-03-18 07:53:46,319::task::568::TaskManager.Task::(_updateState) Task=`18892297-8024-4fae-9e6b-20128f8150d6`::moving from state preparing -> state finished
>Thread-15490::DEBUG::2013-03-18 07:53:46,319::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15490::DEBUG::2013-03-18 07:53:46,319::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15490::DEBUG::2013-03-18 07:53:46,319::task::957::TaskManager.Task::(_decref) Task=`18892297-8024-4fae-9e6b-20128f8150d6`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:53:49,208::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 07:53:49,209::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 07:53:49,210::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:53:49,262::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00806443 s, 254 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:53:49,263::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00806443 s, 254 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 07:53:49,264::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 07:53:49,264::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 07:53:49,274::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 07:53:49,275::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:53:49,275::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:53:49,276::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:53:49,276::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:53:49,277::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:53:49,278::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:53:49,741::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:53:49,746::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:53:49,747::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'POOL_SPM_ID=1', '_SHA_CKSUM=506685b0a94aa1699b56aff09664faa6587ccbb4', 'POOL_SPM_LVER=0']
>Thread-20::WARNING::2013-03-18 07:53:49,749::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 07:53:49,751::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 07:53:49,752::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 07:53:49,753::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:53:49,975::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:53:50,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00210355 s, 1.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:53:50,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:53:50,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:53:50,195::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0200017 s, 205 kB/s\n'; <rc> = 0
>Thread-15493::DEBUG::2013-03-18 07:53:50,507::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15493::DEBUG::2013-03-18 07:53:50,521::task::568::TaskManager.Task::(_updateState) Task=`2459a378-58d3-452c-a2ed-0a6411fe5c82`::moving from state init -> state preparing
>Thread-15493::INFO::2013-03-18 07:53:50,524::logUtils::37::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=3, spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', conList=[{'connection': '10.34.1.214', 'iqn': 'iqn.2003-01.org.linux-iscsi.dhcp-1-214.x8664:sn.113e745f4931', 'portal': '1', 'user': '', 'password': '******', 'id': '76e6e082-bc2a-47cb-8062-fde36badf398', 'port': '3260'}], options=None)
>Thread-15493::DEBUG::2013-03-18 07:53:50,526::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/iscsiadm -m node -T iqn.2003-01.org.linux-iscsi.dhcp-1-214.x8664:sn.113e745f4931 -I default -p 10.34.1.214:3260 --op=new' (cwd None)
>Thread-15493::DEBUG::2013-03-18 07:53:50,578::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-15493::DEBUG::2013-03-18 07:53:50,580::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/iscsiadm -m node -T iqn.2003-01.org.linux-iscsi.dhcp-1-214.x8664:sn.113e745f4931 -I default -p 10.34.1.214:3260 -l' (cwd None)
>Thread-15493::DEBUG::2013-03-18 07:53:50,632::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-15493::DEBUG::2013-03-18 07:53:50,635::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/iscsiadm -m node -T iqn.2003-01.org.linux-iscsi.dhcp-1-214.x8664:sn.113e745f4931 -I default -p 10.34.1.214:3260 -n node.startup -v manual --op=update' (cwd None)
>Thread-15493::DEBUG::2013-03-18 07:53:50,685::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-15493::INFO::2013-03-18 07:53:50,687::logUtils::39::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': '76e6e082-bc2a-47cb-8062-fde36badf398'}]}
>Thread-15493::DEBUG::2013-03-18 07:53:50,688::task::1151::TaskManager.Task::(prepare) Task=`2459a378-58d3-452c-a2ed-0a6411fe5c82`::finished: {'statuslist': [{'status': 0, 'id': '76e6e082-bc2a-47cb-8062-fde36badf398'}]}
>Thread-15493::DEBUG::2013-03-18 07:53:50,688::task::568::TaskManager.Task::(_updateState) Task=`2459a378-58d3-452c-a2ed-0a6411fe5c82`::moving from state preparing -> state finished
>Thread-15493::DEBUG::2013-03-18 07:53:50,689::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15493::DEBUG::2013-03-18 07:53:50,689::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15493::DEBUG::2013-03-18 07:53:50,689::task::957::TaskManager.Task::(_decref) Task=`2459a378-58d3-452c-a2ed-0a6411fe5c82`::ref 0 aborting False
>Thread-21::DEBUG::2013-03-18 07:53:55,029::misc::1025::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-21::DEBUG::2013-03-18 07:53:55,029::misc::1027::SamplingMethod::(__call__) Got in to sampling method
>Thread-21::DEBUG::2013-03-18 07:53:55,030::misc::1025::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-21::DEBUG::2013-03-18 07:53:55,030::misc::1027::SamplingMethod::(__call__) Got in to sampling method
>Thread-21::DEBUG::2013-03-18 07:53:55,030::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:53:55,117::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:53:55,120::misc::1035::SamplingMethod::(__call__) Returning last result
>MainProcess|Thread-21::DEBUG::2013-03-18 07:53:55,124::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd of=/sys/class/scsi_host/host0/scan' (cwd None)
>MainProcess|Thread-21::DEBUG::2013-03-18 07:53:55,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd of=/sys/class/scsi_host/host1/scan' (cwd None)
>MainProcess|Thread-21::DEBUG::2013-03-18 07:53:55,160::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd of=/sys/class/scsi_host/host2/scan' (cwd None)
>MainProcess|Thread-21::DEBUG::2013-03-18 07:53:55,180::iscsi::388::Storage.ISCSI::(forceIScsiScan) Performing SCSI scan, this will take up to 30 seconds
>Thread-15497::DEBUG::2013-03-18 07:53:56,598::task::568::TaskManager.Task::(_updateState) Task=`4d1fd05b-90d8-4077-83ad-331cc2cfe096`::moving from state init -> state preparing
>Thread-15497::INFO::2013-03-18 07:53:56,604::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15497::INFO::2013-03-18 07:53:56,605::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0618810653687', 'lastCheck': '6.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '11.6', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0549740791321', 'lastCheck': '6.6', 'code': 0, 'valid': True}}
>Thread-15497::DEBUG::2013-03-18 07:53:56,606::task::1151::TaskManager.Task::(prepare) Task=`4d1fd05b-90d8-4077-83ad-331cc2cfe096`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0618810653687', 'lastCheck': '6.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '11.6', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0549740791321', 'lastCheck': '6.6', 'code': 0, 'valid': True}}
>Thread-15497::DEBUG::2013-03-18 07:53:56,606::task::568::TaskManager.Task::(_updateState) Task=`4d1fd05b-90d8-4077-83ad-331cc2cfe096`::moving from state preparing -> state finished
>Thread-15497::DEBUG::2013-03-18 07:53:56,606::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15497::DEBUG::2013-03-18 07:53:56,607::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15497::DEBUG::2013-03-18 07:53:56,607::task::957::TaskManager.Task::(_decref) Task=`4d1fd05b-90d8-4077-83ad-331cc2cfe096`::ref 0 aborting False
>Thread-21::DEBUG::2013-03-18 07:53:57,187::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:53:57,313::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:53:57,315::lvm::482::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:53:57,316::lvm::484::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-21::DEBUG::2013-03-18 07:53:57,316::lvm::493::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:53:57,317::lvm::495::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-21::DEBUG::2013-03-18 07:53:57,317::lvm::513::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:53:57,317::lvm::515::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-21::DEBUG::2013-03-18 07:53:57,318::misc::1035::SamplingMethod::(__call__) Returning last result
>Thread-21::DEBUG::2013-03-18 07:53:57,318::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:53:57,324::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:53:57,806::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:53:57,812::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:53:57,814::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:54:00,038::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:54:00,040::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:54:00,236::lvm::373::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-20::DEBUG::2013-03-18 07:54:00,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:54:00,562::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:54:00,566::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:54:00,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0179917 s, 228 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:54:00,821::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:54:00,824::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:54:00,825::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:54:00,872::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0161453 s, 254 kB/s\n'; <rc> = 0
>Thread-15504::DEBUG::2013-03-18 07:54:06,848::task::568::TaskManager.Task::(_updateState) Task=`d71030be-f67f-4819-8f29-fcd31e12777c`::moving from state init -> state preparing
>Thread-15504::INFO::2013-03-18 07:54:06,849::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15504::INFO::2013-03-18 07:54:06,850::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0482549667358', 'lastCheck': '5.9', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '21.8', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0655701160431', 'lastCheck': '6.2', 'code': 0, 'valid': True}}
>Thread-15504::DEBUG::2013-03-18 07:54:06,850::task::1151::TaskManager.Task::(prepare) Task=`d71030be-f67f-4819-8f29-fcd31e12777c`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0482549667358', 'lastCheck': '5.9', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '21.8', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0655701160431', 'lastCheck': '6.2', 'code': 0, 'valid': True}}
>Thread-15504::DEBUG::2013-03-18 07:54:06,851::task::568::TaskManager.Task::(_updateState) Task=`d71030be-f67f-4819-8f29-fcd31e12777c`::moving from state preparing -> state finished
>Thread-15504::DEBUG::2013-03-18 07:54:06,851::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15504::DEBUG::2013-03-18 07:54:06,851::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15504::DEBUG::2013-03-18 07:54:06,853::task::957::TaskManager.Task::(_decref) Task=`d71030be-f67f-4819-8f29-fcd31e12777c`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:54:10,636::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:54:10,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0009103 s, 4.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:54:10,912::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:54:10,993::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.012818 s, 320 kB/s\n'; <rc> = 0
>Thread-15510::DEBUG::2013-03-18 07:54:17,091::task::568::TaskManager.Task::(_updateState) Task=`4da7b80f-3c12-42c7-820d-2d3a8246a75c`::moving from state init -> state preparing
>Thread-15510::INFO::2013-03-18 07:54:17,092::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15510::INFO::2013-03-18 07:54:17,092::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0820510387421', 'lastCheck': '6.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '32.1', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0358738899231', 'lastCheck': '6.4', 'code': 0, 'valid': True}}
>Thread-15510::DEBUG::2013-03-18 07:54:17,093::task::1151::TaskManager.Task::(prepare) Task=`4da7b80f-3c12-42c7-820d-2d3a8246a75c`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0820510387421', 'lastCheck': '6.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '32.1', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0358738899231', 'lastCheck': '6.4', 'code': 0, 'valid': True}}
>Thread-15510::DEBUG::2013-03-18 07:54:17,093::task::568::TaskManager.Task::(_updateState) Task=`4da7b80f-3c12-42c7-820d-2d3a8246a75c`::moving from state preparing -> state finished
>Thread-15510::DEBUG::2013-03-18 07:54:17,093::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15510::DEBUG::2013-03-18 07:54:17,093::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15510::DEBUG::2013-03-18 07:54:17,094::task::957::TaskManager.Task::(_decref) Task=`4da7b80f-3c12-42c7-820d-2d3a8246a75c`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:54:20,676::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:54:20,713::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147119 s, 2.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:54:21,037::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:54:21,073::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0049827 s, 822 kB/s\n'; <rc> = 0
>Thread-15516::DEBUG::2013-03-18 07:54:27,358::task::568::TaskManager.Task::(_updateState) Task=`f1b98e22-a7b0-4d7a-bc5d-b5f18c60f28f`::moving from state init -> state preparing
>Thread-15516::INFO::2013-03-18 07:54:27,359::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15516::INFO::2013-03-18 07:54:27,360::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0375339984894', 'lastCheck': '6.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '42.3', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0388429164886', 'lastCheck': '6.6', 'code': 0, 'valid': True}}
>Thread-15516::DEBUG::2013-03-18 07:54:27,360::task::1151::TaskManager.Task::(prepare) Task=`f1b98e22-a7b0-4d7a-bc5d-b5f18c60f28f`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0375339984894', 'lastCheck': '6.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '42.3', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0388429164886', 'lastCheck': '6.6', 'code': 0, 'valid': True}}
>Thread-15516::DEBUG::2013-03-18 07:54:27,360::task::568::TaskManager.Task::(_updateState) Task=`f1b98e22-a7b0-4d7a-bc5d-b5f18c60f28f`::moving from state preparing -> state finished
>Thread-15516::DEBUG::2013-03-18 07:54:27,361::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15516::DEBUG::2013-03-18 07:54:27,361::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15516::DEBUG::2013-03-18 07:54:27,361::task::957::TaskManager.Task::(_decref) Task=`f1b98e22-a7b0-4d7a-bc5d-b5f18c60f28f`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:54:30,717::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:54:30,759::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159094 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:54:31,114::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:54:31,159::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0140568 s, 291 kB/s\n'; <rc> = 0
>Thread-15522::DEBUG::2013-03-18 07:54:37,584::task::568::TaskManager.Task::(_updateState) Task=`b3e37337-2b7e-4507-aaed-bd86aeff83dd`::moving from state init -> state preparing
>Thread-15522::INFO::2013-03-18 07:54:37,585::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15522::INFO::2013-03-18 07:54:37,586::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0485589504242', 'lastCheck': '6.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '52.6', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0432150363922', 'lastCheck': '6.8', 'code': 0, 'valid': True}}
>Thread-15522::DEBUG::2013-03-18 07:54:37,586::task::1151::TaskManager.Task::(prepare) Task=`b3e37337-2b7e-4507-aaed-bd86aeff83dd`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0485589504242', 'lastCheck': '6.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '52.6', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0432150363922', 'lastCheck': '6.8', 'code': 0, 'valid': True}}
>Thread-15522::DEBUG::2013-03-18 07:54:37,586::task::568::TaskManager.Task::(_updateState) Task=`b3e37337-2b7e-4507-aaed-bd86aeff83dd`::moving from state preparing -> state finished
>Thread-15522::DEBUG::2013-03-18 07:54:37,587::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15522::DEBUG::2013-03-18 07:54:37,587::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15522::DEBUG::2013-03-18 07:54:37,587::task::957::TaskManager.Task::(_decref) Task=`b3e37337-2b7e-4507-aaed-bd86aeff83dd`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:54:40,764::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:54:40,818::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000945857 s, 4.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:54:41,201::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:54:41,281::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0327524 s, 125 kB/s\n'; <rc> = 0
>Thread-15528::DEBUG::2013-03-18 07:54:47,798::task::568::TaskManager.Task::(_updateState) Task=`1d0ce3af-4c42-4bb2-b718-e4a7989f4bc3`::moving from state init -> state preparing
>Thread-15528::INFO::2013-03-18 07:54:47,799::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15528::INFO::2013-03-18 07:54:47,799::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0825409889221', 'lastCheck': '6.5', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '62.8', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0557022094727', 'lastCheck': '7.0', 'code': 0, 'valid': True}}
>Thread-15528::DEBUG::2013-03-18 07:54:47,800::task::1151::TaskManager.Task::(prepare) Task=`1d0ce3af-4c42-4bb2-b718-e4a7989f4bc3`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0825409889221', 'lastCheck': '6.5', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '62.8', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0557022094727', 'lastCheck': '7.0', 'code': 0, 'valid': True}}
>Thread-15528::DEBUG::2013-03-18 07:54:47,800::task::568::TaskManager.Task::(_updateState) Task=`1d0ce3af-4c42-4bb2-b718-e4a7989f4bc3`::moving from state preparing -> state finished
>Thread-15528::DEBUG::2013-03-18 07:54:47,800::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15528::DEBUG::2013-03-18 07:54:47,800::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15528::DEBUG::2013-03-18 07:54:47,801::task::957::TaskManager.Task::(_decref) Task=`1d0ce3af-4c42-4bb2-b718-e4a7989f4bc3`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:54:50,822::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:54:50,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0032193 s, 1.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:54:51,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:54:51,384::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0115552 s, 354 kB/s\n'; <rc> = 0
>Thread-15499::WARNING::2013-03-18 07:54:57,843::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:54:57,847::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-15534::DEBUG::2013-03-18 07:54:58,029::task::568::TaskManager.Task::(_updateState) Task=`32aa5c99-7dc5-4c84-a80d-6af998fb7e3d`::moving from state init -> state preparing
>Thread-15534::INFO::2013-03-18 07:54:58,030::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15534::INFO::2013-03-18 07:54:58,031::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0580449104309', 'lastCheck': '6.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '0.2', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0373618602753', 'lastCheck': '7.2', 'code': 0, 'valid': True}}
>Thread-15534::DEBUG::2013-03-18 07:54:58,031::task::1151::TaskManager.Task::(prepare) Task=`32aa5c99-7dc5-4c84-a80d-6af998fb7e3d`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0580449104309', 'lastCheck': '6.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '0.2', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0373618602753', 'lastCheck': '7.2', 'code': 0, 'valid': True}}
>Thread-15534::DEBUG::2013-03-18 07:54:58,032::task::568::TaskManager.Task::(_updateState) Task=`32aa5c99-7dc5-4c84-a80d-6af998fb7e3d`::moving from state preparing -> state finished
>Thread-15534::DEBUG::2013-03-18 07:54:58,032::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15534::DEBUG::2013-03-18 07:54:58,033::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15534::DEBUG::2013-03-18 07:54:58,033::task::957::TaskManager.Task::(_decref) Task=`32aa5c99-7dc5-4c84-a80d-6af998fb7e3d`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:55:00,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:55:00,908::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175899 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:55:01,430::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:55:01,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186802 s, 219 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:55:07,850::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:55:07,851::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-15540::DEBUG::2013-03-18 07:55:08,291::task::568::TaskManager.Task::(_updateState) Task=`b4f74126-4c4d-42c7-8a63-1db97ed178af`::moving from state init -> state preparing
>Thread-15540::INFO::2013-03-18 07:55:08,293::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15540::INFO::2013-03-18 07:55:08,294::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0521891117096', 'lastCheck': '6.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '10.4', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0455849170685', 'lastCheck': '7.4', 'code': 0, 'valid': True}}
>Thread-15540::DEBUG::2013-03-18 07:55:08,294::task::1151::TaskManager.Task::(prepare) Task=`b4f74126-4c4d-42c7-8a63-1db97ed178af`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0521891117096', 'lastCheck': '6.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '10.4', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0455849170685', 'lastCheck': '7.4', 'code': 0, 'valid': True}}
>Thread-15540::DEBUG::2013-03-18 07:55:08,295::task::568::TaskManager.Task::(_updateState) Task=`b4f74126-4c4d-42c7-8a63-1db97ed178af`::moving from state preparing -> state finished
>Thread-15540::DEBUG::2013-03-18 07:55:08,295::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15540::DEBUG::2013-03-18 07:55:08,295::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15540::DEBUG::2013-03-18 07:55:08,296::task::957::TaskManager.Task::(_decref) Task=`b4f74126-4c4d-42c7-8a63-1db97ed178af`::ref 0 aborting False
>Thread-21::DEBUG::2013-03-18 07:55:08,393::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:55:08,400::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:55:08,402::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:55:10,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:55:10,957::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00155779 s, 2.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:55:11,522::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:55:11,561::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0113808 s, 360 kB/s\n'; <rc> = 0
>Thread-15547::DEBUG::2013-03-18 07:55:18,693::task::568::TaskManager.Task::(_updateState) Task=`c7883f03-113d-4d1c-862e-85a8c0563f70`::moving from state init -> state preparing
>Thread-15547::INFO::2013-03-18 07:55:18,694::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15547::INFO::2013-03-18 07:55:18,695::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0409231185913', 'lastCheck': '7.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '20.8', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0452868938446', 'lastCheck': '7.7', 'code': 0, 'valid': True}}
>Thread-15547::DEBUG::2013-03-18 07:55:18,695::task::1151::TaskManager.Task::(prepare) Task=`c7883f03-113d-4d1c-862e-85a8c0563f70`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0409231185913', 'lastCheck': '7.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '20.8', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0452868938446', 'lastCheck': '7.7', 'code': 0, 'valid': True}}
>Thread-15547::DEBUG::2013-03-18 07:55:18,695::task::568::TaskManager.Task::(_updateState) Task=`c7883f03-113d-4d1c-862e-85a8c0563f70`::moving from state preparing -> state finished
>Thread-15547::DEBUG::2013-03-18 07:55:18,696::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15547::DEBUG::2013-03-18 07:55:18,696::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15547::DEBUG::2013-03-18 07:55:18,696::task::957::TaskManager.Task::(_decref) Task=`c7883f03-113d-4d1c-862e-85a8c0563f70`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:55:20,962::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:55:20,998::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00175809 s, 2.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:55:21,601::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:55:21,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0123383 s, 332 kB/s\n'; <rc> = 0
>Thread-15553::DEBUG::2013-03-18 07:55:28,951::task::568::TaskManager.Task::(_updateState) Task=`bba3cc9b-ce53-44e6-875e-483bfcf5f3f6`::moving from state init -> state preparing
>Thread-15553::INFO::2013-03-18 07:55:28,952::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15553::INFO::2013-03-18 07:55:28,953::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0799391269684', 'lastCheck': '7.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '31.1', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0380170345306', 'lastCheck': '8.0', 'code': 0, 'valid': True}}
>Thread-15553::DEBUG::2013-03-18 07:55:28,953::task::1151::TaskManager.Task::(prepare) Task=`bba3cc9b-ce53-44e6-875e-483bfcf5f3f6`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0799391269684', 'lastCheck': '7.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '31.1', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0380170345306', 'lastCheck': '8.0', 'code': 0, 'valid': True}}
>Thread-15553::DEBUG::2013-03-18 07:55:28,953::task::568::TaskManager.Task::(_updateState) Task=`bba3cc9b-ce53-44e6-875e-483bfcf5f3f6`::moving from state preparing -> state finished
>Thread-15553::DEBUG::2013-03-18 07:55:28,953::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15553::DEBUG::2013-03-18 07:55:28,954::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15553::DEBUG::2013-03-18 07:55:28,954::task::957::TaskManager.Task::(_decref) Task=`bba3cc9b-ce53-44e6-875e-483bfcf5f3f6`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:55:31,003::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:55:31,053::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00101934 s, 4.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:55:31,735::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:55:31,796::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159236 s, 257 kB/s\n'; <rc> = 0
>Thread-15559::DEBUG::2013-03-18 07:55:39,210::task::568::TaskManager.Task::(_updateState) Task=`319b5d6f-25fa-4b44-bc49-1f92d60eb8be`::moving from state init -> state preparing
>Thread-15559::INFO::2013-03-18 07:55:39,212::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15559::INFO::2013-03-18 07:55:39,213::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0630328655243', 'lastCheck': '7.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '41.4', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0533978939056', 'lastCheck': '8.1', 'code': 0, 'valid': True}}
>Thread-15559::DEBUG::2013-03-18 07:55:39,213::task::1151::TaskManager.Task::(prepare) Task=`319b5d6f-25fa-4b44-bc49-1f92d60eb8be`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0630328655243', 'lastCheck': '7.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '41.4', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0533978939056', 'lastCheck': '8.1', 'code': 0, 'valid': True}}
>Thread-15559::DEBUG::2013-03-18 07:55:39,213::task::568::TaskManager.Task::(_updateState) Task=`319b5d6f-25fa-4b44-bc49-1f92d60eb8be`::moving from state preparing -> state finished
>Thread-15559::DEBUG::2013-03-18 07:55:39,214::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15559::DEBUG::2013-03-18 07:55:39,214::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15559::DEBUG::2013-03-18 07:55:39,214::task::957::TaskManager.Task::(_decref) Task=`319b5d6f-25fa-4b44-bc49-1f92d60eb8be`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:55:41,064::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:55:41,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000869889 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:55:41,836::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:55:41,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00636834 s, 643 kB/s\n'; <rc> = 0
>Thread-15565::DEBUG::2013-03-18 07:55:49,469::task::568::TaskManager.Task::(_updateState) Task=`58a8854f-7fb6-49b0-ad1c-66b585b8b70a`::moving from state init -> state preparing
>Thread-15565::INFO::2013-03-18 07:55:49,469::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15565::INFO::2013-03-18 07:55:49,470::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0449478626251', 'lastCheck': '7.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '51.6', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0330538749695', 'lastCheck': '8.4', 'code': 0, 'valid': True}}
>Thread-15565::DEBUG::2013-03-18 07:55:49,471::task::1151::TaskManager.Task::(prepare) Task=`58a8854f-7fb6-49b0-ad1c-66b585b8b70a`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0449478626251', 'lastCheck': '7.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '51.6', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0330538749695', 'lastCheck': '8.4', 'code': 0, 'valid': True}}
>Thread-15565::DEBUG::2013-03-18 07:55:49,471::task::568::TaskManager.Task::(_updateState) Task=`58a8854f-7fb6-49b0-ad1c-66b585b8b70a`::moving from state preparing -> state finished
>Thread-15565::DEBUG::2013-03-18 07:55:49,471::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15565::DEBUG::2013-03-18 07:55:49,471::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15565::DEBUG::2013-03-18 07:55:49,472::task::957::TaskManager.Task::(_decref) Task=`58a8854f-7fb6-49b0-ad1c-66b585b8b70a`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:55:51,100::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:55:51,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000742666 s, 5.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:55:51,920::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:55:51,953::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00769966 s, 532 kB/s\n'; <rc> = 0
>Thread-15571::DEBUG::2013-03-18 07:55:59,686::task::568::TaskManager.Task::(_updateState) Task=`635ae3fe-5502-48df-8cc5-30e0b951fac0`::moving from state init -> state preparing
>Thread-15571::INFO::2013-03-18 07:55:59,687::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15571::INFO::2013-03-18 07:55:59,687::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0347490310669', 'lastCheck': '7.7', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '61.8', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.031996011734', 'lastCheck': '8.6', 'code': 0, 'valid': True}}
>Thread-15571::DEBUG::2013-03-18 07:55:59,688::task::1151::TaskManager.Task::(prepare) Task=`635ae3fe-5502-48df-8cc5-30e0b951fac0`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0347490310669', 'lastCheck': '7.7', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '61.8', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.031996011734', 'lastCheck': '8.6', 'code': 0, 'valid': True}}
>Thread-15571::DEBUG::2013-03-18 07:55:59,688::task::568::TaskManager.Task::(_updateState) Task=`635ae3fe-5502-48df-8cc5-30e0b951fac0`::moving from state preparing -> state finished
>Thread-15571::DEBUG::2013-03-18 07:55:59,688::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15571::DEBUG::2013-03-18 07:55:59,688::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15571::DEBUG::2013-03-18 07:55:59,688::task::957::TaskManager.Task::(_decref) Task=`635ae3fe-5502-48df-8cc5-30e0b951fac0`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:56:01,135::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:56:01,201::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000881615 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:56:01,998::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:56:02,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000845211 s, 4.8 MB/s\n'; <rc> = 0
>Thread-15542::WARNING::2013-03-18 07:56:08,456::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:56:08,460::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-15577::DEBUG::2013-03-18 07:56:09,943::task::568::TaskManager.Task::(_updateState) Task=`4d52091a-3094-42e7-a267-65a772f3f74b`::moving from state init -> state preparing
>Thread-15577::INFO::2013-03-18 07:56:09,944::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15577::INFO::2013-03-18 07:56:09,944::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.045872926712', 'lastCheck': '7.9', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '1.5', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.067538022995', 'lastCheck': '8.7', 'code': 0, 'valid': True}}
>Thread-15577::DEBUG::2013-03-18 07:56:09,945::task::1151::TaskManager.Task::(prepare) Task=`4d52091a-3094-42e7-a267-65a772f3f74b`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.045872926712', 'lastCheck': '7.9', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '1.5', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.067538022995', 'lastCheck': '8.7', 'code': 0, 'valid': True}}
>Thread-15577::DEBUG::2013-03-18 07:56:09,945::task::568::TaskManager.Task::(_updateState) Task=`4d52091a-3094-42e7-a267-65a772f3f74b`::moving from state preparing -> state finished
>Thread-15577::DEBUG::2013-03-18 07:56:09,945::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15577::DEBUG::2013-03-18 07:56:09,946::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15577::DEBUG::2013-03-18 07:56:09,946::task::957::TaskManager.Task::(_decref) Task=`4d52091a-3094-42e7-a267-65a772f3f74b`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:56:11,216::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:56:11,247::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000887922 s, 4.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:56:12,089::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:56:12,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753328 s, 5.4 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:56:18,462::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:56:18,463::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:56:19,012::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:56:19,018::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:56:19,019::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-15584::DEBUG::2013-03-18 07:56:20,220::task::568::TaskManager.Task::(_updateState) Task=`6b666892-e99b-477f-bc27-1762a114add4`::moving from state init -> state preparing
>Thread-15584::INFO::2013-03-18 07:56:20,221::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15584::INFO::2013-03-18 07:56:20,222::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0272629261017', 'lastCheck': '8.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '11.8', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0324468612671', 'lastCheck': '9.0', 'code': 0, 'valid': True}}
>Thread-15584::DEBUG::2013-03-18 07:56:20,222::task::1151::TaskManager.Task::(prepare) Task=`6b666892-e99b-477f-bc27-1762a114add4`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0272629261017', 'lastCheck': '8.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '11.8', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0324468612671', 'lastCheck': '9.0', 'code': 0, 'valid': True}}
>Thread-15584::DEBUG::2013-03-18 07:56:20,223::task::568::TaskManager.Task::(_updateState) Task=`6b666892-e99b-477f-bc27-1762a114add4`::moving from state preparing -> state finished
>Thread-15584::DEBUG::2013-03-18 07:56:20,223::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15584::DEBUG::2013-03-18 07:56:20,223::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15584::DEBUG::2013-03-18 07:56:20,223::task::957::TaskManager.Task::(_decref) Task=`6b666892-e99b-477f-bc27-1762a114add4`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:56:21,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:56:21,286::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00087525 s, 4.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:56:22,155::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:56:22,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0159754 s, 256 kB/s\n'; <rc> = 0
>Thread-15590::DEBUG::2013-03-18 07:56:30,615::task::568::TaskManager.Task::(_updateState) Task=`5dd64b6f-1483-4576-935b-effb10bfddd6`::moving from state init -> state preparing
>Thread-15590::INFO::2013-03-18 07:56:30,616::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15590::INFO::2013-03-18 07:56:30,616::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0467090606689', 'lastCheck': '8.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '22.2', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0325510501862', 'lastCheck': '9.3', 'code': 0, 'valid': True}}
>Thread-15590::DEBUG::2013-03-18 07:56:30,616::task::1151::TaskManager.Task::(prepare) Task=`5dd64b6f-1483-4576-935b-effb10bfddd6`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0467090606689', 'lastCheck': '8.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '22.2', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0325510501862', 'lastCheck': '9.3', 'code': 0, 'valid': True}}
>Thread-15590::DEBUG::2013-03-18 07:56:30,617::task::568::TaskManager.Task::(_updateState) Task=`5dd64b6f-1483-4576-935b-effb10bfddd6`::moving from state preparing -> state finished
>Thread-15590::DEBUG::2013-03-18 07:56:30,617::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15590::DEBUG::2013-03-18 07:56:30,617::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15590::DEBUG::2013-03-18 07:56:30,617::task::957::TaskManager.Task::(_decref) Task=`5dd64b6f-1483-4576-935b-effb10bfddd6`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:56:31,294::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:56:31,324::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000809347 s, 5.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:56:32,240::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:56:32,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000862095 s, 4.8 MB/s\n'; <rc> = 0
>Thread-15595::DEBUG::2013-03-18 07:56:38,454::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15595::DEBUG::2013-03-18 07:56:38,455::task::568::TaskManager.Task::(_updateState) Task=`8ff2b1d6-c3d4-4545-89cf-2d162746745d`::moving from state init -> state preparing
>Thread-15595::INFO::2013-03-18 07:56:38,456::logUtils::37::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', hostID=2, scsiKey='a1112cc0-b4af-4396-a3b6-2a3916293a6b', msdUUID='4bf968ef-020a-47ee-a1ae-f85eaeb8751a', masterVersion=1, options=None)
>Thread-15595::DEBUG::2013-03-18 07:56:38,457::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`7e18297f-52e3-4b87-84d2-5d19bb222846`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15595::DEBUG::2013-03-18 07:56:38,457::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-15595::DEBUG::2013-03-18 07:56:38,458::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15595::DEBUG::2013-03-18 07:56:38,458::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`7e18297f-52e3-4b87-84d2-5d19bb222846`::Granted request
>Thread-15595::DEBUG::2013-03-18 07:56:38,460::sp::1536::Storage.StoragePool::(getMasterDomain) Master domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a verified, version 1
>Thread-15595::DEBUG::2013-03-18 07:56:38,460::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15595::DEBUG::2013-03-18 07:56:38,460::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15595::DEBUG::2013-03-18 07:56:38,461::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15595::DEBUG::2013-03-18 07:56:38,461::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15595::INFO::2013-03-18 07:56:38,461::logUtils::39::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: None
>Thread-15595::DEBUG::2013-03-18 07:56:38,461::task::1151::TaskManager.Task::(prepare) Task=`8ff2b1d6-c3d4-4545-89cf-2d162746745d`::finished: None
>Thread-15595::DEBUG::2013-03-18 07:56:38,461::task::568::TaskManager.Task::(_updateState) Task=`8ff2b1d6-c3d4-4545-89cf-2d162746745d`::moving from state preparing -> state finished
>Thread-15595::DEBUG::2013-03-18 07:56:38,462::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15595::DEBUG::2013-03-18 07:56:38,462::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15595::DEBUG::2013-03-18 07:56:38,462::task::957::TaskManager.Task::(_decref) Task=`8ff2b1d6-c3d4-4545-89cf-2d162746745d`::ref 0 aborting False
>Thread-15597::DEBUG::2013-03-18 07:56:40,887::task::568::TaskManager.Task::(_updateState) Task=`db68389d-1852-4e6c-b421-a3349b916c65`::moving from state init -> state preparing
>Thread-15597::INFO::2013-03-18 07:56:40,887::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15597::INFO::2013-03-18 07:56:40,888::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0264918804169', 'lastCheck': '8.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '32.4', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0315599441528', 'lastCheck': '9.6', 'code': 0, 'valid': True}}
>Thread-15597::DEBUG::2013-03-18 07:56:40,888::task::1151::TaskManager.Task::(prepare) Task=`db68389d-1852-4e6c-b421-a3349b916c65`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0264918804169', 'lastCheck': '8.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '32.4', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0315599441528', 'lastCheck': '9.6', 'code': 0, 'valid': True}}
>Thread-15597::DEBUG::2013-03-18 07:56:40,889::task::568::TaskManager.Task::(_updateState) Task=`db68389d-1852-4e6c-b421-a3349b916c65`::moving from state preparing -> state finished
>Thread-15597::DEBUG::2013-03-18 07:56:40,889::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15597::DEBUG::2013-03-18 07:56:40,889::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15597::DEBUG::2013-03-18 07:56:40,889::task::957::TaskManager.Task::(_decref) Task=`db68389d-1852-4e6c-b421-a3349b916c65`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:56:41,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:56:41,376::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00140216 s, 2.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:56:42,307::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:56:42,356::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00640418 s, 640 kB/s\n'; <rc> = 0
>Thread-15603::DEBUG::2013-03-18 07:56:51,140::task::568::TaskManager.Task::(_updateState) Task=`52308c0d-8856-4a89-aef1-2d8471341de3`::moving from state init -> state preparing
>Thread-15603::INFO::2013-03-18 07:56:51,141::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15603::INFO::2013-03-18 07:56:51,141::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0523159503937', 'lastCheck': '8.7', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '42.7', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0483379364014', 'lastCheck': '9.8', 'code': 0, 'valid': True}}
>Thread-15603::DEBUG::2013-03-18 07:56:51,142::task::1151::TaskManager.Task::(prepare) Task=`52308c0d-8856-4a89-aef1-2d8471341de3`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0523159503937', 'lastCheck': '8.7', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '42.7', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0483379364014', 'lastCheck': '9.8', 'code': 0, 'valid': True}}
>Thread-15603::DEBUG::2013-03-18 07:56:51,142::task::568::TaskManager.Task::(_updateState) Task=`52308c0d-8856-4a89-aef1-2d8471341de3`::moving from state preparing -> state finished
>Thread-15603::DEBUG::2013-03-18 07:56:51,142::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15603::DEBUG::2013-03-18 07:56:51,143::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15603::DEBUG::2013-03-18 07:56:51,143::task::957::TaskManager.Task::(_decref) Task=`52308c0d-8856-4a89-aef1-2d8471341de3`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:56:51,381::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:56:51,423::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00132221 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:56:52,400::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:56:52,448::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0185869 s, 220 kB/s\n'; <rc> = 0
>Thread-15609::DEBUG::2013-03-18 07:57:01,375::task::568::TaskManager.Task::(_updateState) Task=`19012f7d-e2b3-4eba-9a60-aa74542de4bd`::moving from state init -> state preparing
>Thread-15609::INFO::2013-03-18 07:57:01,376::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15609::INFO::2013-03-18 07:57:01,377::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0512299537659', 'lastCheck': '8.9', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '52.9', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0437619686127', 'lastCheck': '10.0', 'code': 0, 'valid': True}}
>Thread-15609::DEBUG::2013-03-18 07:57:01,377::task::1151::TaskManager.Task::(prepare) Task=`19012f7d-e2b3-4eba-9a60-aa74542de4bd`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0512299537659', 'lastCheck': '8.9', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '52.9', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0437619686127', 'lastCheck': '10.0', 'code': 0, 'valid': True}}
>Thread-15609::DEBUG::2013-03-18 07:57:01,377::task::568::TaskManager.Task::(_updateState) Task=`19012f7d-e2b3-4eba-9a60-aa74542de4bd`::moving from state preparing -> state finished
>Thread-15609::DEBUG::2013-03-18 07:57:01,378::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15609::DEBUG::2013-03-18 07:57:01,378::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15609::DEBUG::2013-03-18 07:57:01,378::task::957::TaskManager.Task::(_decref) Task=`19012f7d-e2b3-4eba-9a60-aa74542de4bd`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:57:01,428::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:57:01,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00138635 s, 3.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:57:02,491::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:57:02,548::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0213284 s, 192 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:57:11,474::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:57:11,514::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00150338 s, 2.7 MB/s\n'; <rc> = 0
>Thread-15615::DEBUG::2013-03-18 07:57:11,628::task::568::TaskManager.Task::(_updateState) Task=`11e0dbe6-a8f5-4f5a-af80-5b80eac7ed16`::moving from state init -> state preparing
>Thread-15615::INFO::2013-03-18 07:57:11,629::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15615::INFO::2013-03-18 07:57:11,630::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0603859424591', 'lastCheck': '9.0', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '63.2', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0420529842377', 'lastCheck': '0.1', 'code': 0, 'valid': True}}
>Thread-15615::DEBUG::2013-03-18 07:57:11,630::task::1151::TaskManager.Task::(prepare) Task=`11e0dbe6-a8f5-4f5a-af80-5b80eac7ed16`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0603859424591', 'lastCheck': '9.0', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '63.2', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0420529842377', 'lastCheck': '0.1', 'code': 0, 'valid': True}}
>Thread-15615::DEBUG::2013-03-18 07:57:11,630::task::568::TaskManager.Task::(_updateState) Task=`11e0dbe6-a8f5-4f5a-af80-5b80eac7ed16`::moving from state preparing -> state finished
>Thread-15615::DEBUG::2013-03-18 07:57:11,631::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15615::DEBUG::2013-03-18 07:57:11,631::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15615::DEBUG::2013-03-18 07:57:11,631::task::957::TaskManager.Task::(_decref) Task=`11e0dbe6-a8f5-4f5a-af80-5b80eac7ed16`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:57:12,590::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:57:12,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0164037 s, 250 kB/s\n'; <rc> = 0
>Thread-15583::WARNING::2013-03-18 07:57:19,058::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:57:19,061::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 07:57:21,519::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:57:21,548::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000827468 s, 5.0 MB/s\n'; <rc> = 0
>Thread-15621::DEBUG::2013-03-18 07:57:21,899::task::568::TaskManager.Task::(_updateState) Task=`f2d59c32-cd9a-4034-8529-c13c4683b656`::moving from state init -> state preparing
>Thread-15621::INFO::2013-03-18 07:57:21,900::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15621::INFO::2013-03-18 07:57:21,900::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0526530742645', 'lastCheck': '9.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '2.8', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0310821533203', 'lastCheck': '0.3', 'code': 0, 'valid': True}}
>Thread-15621::DEBUG::2013-03-18 07:57:21,900::task::1151::TaskManager.Task::(prepare) Task=`f2d59c32-cd9a-4034-8529-c13c4683b656`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0526530742645', 'lastCheck': '9.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '2.8', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0310821533203', 'lastCheck': '0.3', 'code': 0, 'valid': True}}
>Thread-15621::DEBUG::2013-03-18 07:57:21,901::task::568::TaskManager.Task::(_updateState) Task=`f2d59c32-cd9a-4034-8529-c13c4683b656`::moving from state preparing -> state finished
>Thread-15621::DEBUG::2013-03-18 07:57:21,901::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15621::DEBUG::2013-03-18 07:57:21,901::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15621::DEBUG::2013-03-18 07:57:21,901::task::957::TaskManager.Task::(_decref) Task=`f2d59c32-cd9a-4034-8529-c13c4683b656`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:57:22,684::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:57:22,748::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0328895 s, 125 kB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 07:57:29,063::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:57:29,064::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:57:29,575::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:57:29,582::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:57:29,583::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:57:31,555::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:57:31,583::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00102786 s, 4.0 MB/s\n'; <rc> = 0
>Thread-15628::DEBUG::2013-03-18 07:57:32,178::task::568::TaskManager.Task::(_updateState) Task=`f1f8caea-03d7-47cd-b341-75f5e64515ec`::moving from state init -> state preparing
>Thread-15628::INFO::2013-03-18 07:57:32,179::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15628::INFO::2013-03-18 07:57:32,179::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0661640167236', 'lastCheck': '9.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '13.1', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0300400257111', 'lastCheck': '0.6', 'code': 0, 'valid': True}}
>Thread-15628::DEBUG::2013-03-18 07:57:32,180::task::1151::TaskManager.Task::(prepare) Task=`f1f8caea-03d7-47cd-b341-75f5e64515ec`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0661640167236', 'lastCheck': '9.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '13.1', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0300400257111', 'lastCheck': '0.6', 'code': 0, 'valid': True}}
>Thread-15628::DEBUG::2013-03-18 07:57:32,180::task::568::TaskManager.Task::(_updateState) Task=`f1f8caea-03d7-47cd-b341-75f5e64515ec`::moving from state preparing -> state finished
>Thread-15628::DEBUG::2013-03-18 07:57:32,180::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15628::DEBUG::2013-03-18 07:57:32,181::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15628::DEBUG::2013-03-18 07:57:32,181::task::957::TaskManager.Task::(_decref) Task=`f1f8caea-03d7-47cd-b341-75f5e64515ec`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:57:32,788::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:57:32,829::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0129322 s, 317 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:57:41,588::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:57:41,615::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000753411 s, 5.4 MB/s\n'; <rc> = 0
>Thread-15634::DEBUG::2013-03-18 07:57:42,403::task::568::TaskManager.Task::(_updateState) Task=`43c6963d-b92d-433c-86fa-7f4d74ff9782`::moving from state init -> state preparing
>Thread-15634::INFO::2013-03-18 07:57:42,403::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15634::INFO::2013-03-18 07:57:42,404::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.042661190033', 'lastCheck': '9.5', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '23.3', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0285918712616', 'lastCheck': '0.8', 'code': 0, 'valid': True}}
>Thread-15634::DEBUG::2013-03-18 07:57:42,404::task::1151::TaskManager.Task::(prepare) Task=`43c6963d-b92d-433c-86fa-7f4d74ff9782`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.042661190033', 'lastCheck': '9.5', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '23.3', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0285918712616', 'lastCheck': '0.8', 'code': 0, 'valid': True}}
>Thread-15634::DEBUG::2013-03-18 07:57:42,405::task::568::TaskManager.Task::(_updateState) Task=`43c6963d-b92d-433c-86fa-7f4d74ff9782`::moving from state preparing -> state finished
>Thread-15634::DEBUG::2013-03-18 07:57:42,405::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15634::DEBUG::2013-03-18 07:57:42,405::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15634::DEBUG::2013-03-18 07:57:42,405::task::957::TaskManager.Task::(_decref) Task=`43c6963d-b92d-433c-86fa-7f4d74ff9782`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:57:42,869::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:57:42,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0186467 s, 220 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:57:51,620::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:57:51,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00154228 s, 2.7 MB/s\n'; <rc> = 0
>Thread-15640::DEBUG::2013-03-18 07:57:52,803::task::568::TaskManager.Task::(_updateState) Task=`a2a020ee-bda6-4ff3-8185-46be808229c5`::moving from state init -> state preparing
>Thread-15640::INFO::2013-03-18 07:57:52,804::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15640::INFO::2013-03-18 07:57:52,806::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.063549041748', 'lastCheck': '9.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '33.7', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.045294046402', 'lastCheck': '1.1', 'code': 0, 'valid': True}}
>Thread-15640::DEBUG::2013-03-18 07:57:52,806::task::1151::TaskManager.Task::(prepare) Task=`a2a020ee-bda6-4ff3-8185-46be808229c5`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.063549041748', 'lastCheck': '9.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '33.7', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.045294046402', 'lastCheck': '1.1', 'code': 0, 'valid': True}}
>Thread-15640::DEBUG::2013-03-18 07:57:52,807::task::568::TaskManager.Task::(_updateState) Task=`a2a020ee-bda6-4ff3-8185-46be808229c5`::moving from state preparing -> state finished
>Thread-15640::DEBUG::2013-03-18 07:57:52,807::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15640::DEBUG::2013-03-18 07:57:52,808::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15640::DEBUG::2013-03-18 07:57:52,808::task::957::TaskManager.Task::(_decref) Task=`a2a020ee-bda6-4ff3-8185-46be808229c5`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:57:52,971::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:57:53,002::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000788693 s, 5.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:58:01,669::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:58:01,722::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0170453 s, 240 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:58:03,043::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:58:03,098::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000959496 s, 4.3 MB/s\n'; <rc> = 0
>Thread-15646::DEBUG::2013-03-18 07:58:03,144::task::568::TaskManager.Task::(_updateState) Task=`abaee76b-60a3-4259-bbc5-4f4a43e65225`::moving from state init -> state preparing
>Thread-15646::INFO::2013-03-18 07:58:03,148::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15646::INFO::2013-03-18 07:58:03,152::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.033698797226', 'lastCheck': '10.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '44.1', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0579850673676', 'lastCheck': '1.4', 'code': 0, 'valid': True}}
>Thread-15646::DEBUG::2013-03-18 07:58:03,154::task::1151::TaskManager.Task::(prepare) Task=`abaee76b-60a3-4259-bbc5-4f4a43e65225`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.033698797226', 'lastCheck': '10.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '44.1', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0579850673676', 'lastCheck': '1.4', 'code': 0, 'valid': True}}
>Thread-15646::DEBUG::2013-03-18 07:58:03,155::task::568::TaskManager.Task::(_updateState) Task=`abaee76b-60a3-4259-bbc5-4f4a43e65225`::moving from state preparing -> state finished
>Thread-15646::DEBUG::2013-03-18 07:58:03,157::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15646::DEBUG::2013-03-18 07:58:03,159::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15646::DEBUG::2013-03-18 07:58:03,162::task::957::TaskManager.Task::(_decref) Task=`abaee76b-60a3-4259-bbc5-4f4a43e65225`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:58:11,730::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:58:11,772::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00127217 s, 3.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:58:13,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:58:13,269::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00747935 s, 548 kB/s\n'; <rc> = 0
>Thread-15652::DEBUG::2013-03-18 07:58:13,427::task::568::TaskManager.Task::(_updateState) Task=`1985612e-4126-4bd0-8437-e57d82f317c9`::moving from state init -> state preparing
>Thread-15652::INFO::2013-03-18 07:58:13,428::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15652::INFO::2013-03-18 07:58:13,429::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0548551082611', 'lastCheck': '0.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '54.4', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0435600280762', 'lastCheck': '1.7', 'code': 0, 'valid': True}}
>Thread-15652::DEBUG::2013-03-18 07:58:13,429::task::1151::TaskManager.Task::(prepare) Task=`1985612e-4126-4bd0-8437-e57d82f317c9`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0548551082611', 'lastCheck': '0.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '54.4', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0435600280762', 'lastCheck': '1.7', 'code': 0, 'valid': True}}
>Thread-15652::DEBUG::2013-03-18 07:58:13,429::task::568::TaskManager.Task::(_updateState) Task=`1985612e-4126-4bd0-8437-e57d82f317c9`::moving from state preparing -> state finished
>Thread-15652::DEBUG::2013-03-18 07:58:13,430::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15652::DEBUG::2013-03-18 07:58:13,430::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15652::DEBUG::2013-03-18 07:58:13,430::task::957::TaskManager.Task::(_decref) Task=`1985612e-4126-4bd0-8437-e57d82f317c9`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:58:21,777::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:58:21,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131131 s, 3.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:58:23,321::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:58:23,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00345283 s, 1.2 MB/s\n'; <rc> = 0
>Thread-15658::DEBUG::2013-03-18 07:58:23,705::task::568::TaskManager.Task::(_updateState) Task=`a23d2f0c-6d0c-441c-a8bc-fff6429106dd`::moving from state init -> state preparing
>Thread-15658::INFO::2013-03-18 07:58:23,706::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15658::INFO::2013-03-18 07:58:23,707::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0366191864014', 'lastCheck': '0.3', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '64.6', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0397579669952', 'lastCheck': '1.9', 'code': 0, 'valid': True}}
>Thread-15658::DEBUG::2013-03-18 07:58:23,707::task::1151::TaskManager.Task::(prepare) Task=`a23d2f0c-6d0c-441c-a8bc-fff6429106dd`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0366191864014', 'lastCheck': '0.3', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '64.6', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0397579669952', 'lastCheck': '1.9', 'code': 0, 'valid': True}}
>Thread-15658::DEBUG::2013-03-18 07:58:23,707::task::568::TaskManager.Task::(_updateState) Task=`a23d2f0c-6d0c-441c-a8bc-fff6429106dd`::moving from state preparing -> state finished
>Thread-15658::DEBUG::2013-03-18 07:58:23,708::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15658::DEBUG::2013-03-18 07:58:23,708::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15658::DEBUG::2013-03-18 07:58:23,708::task::957::TaskManager.Task::(_decref) Task=`a23d2f0c-6d0c-441c-a8bc-fff6429106dd`::ref 0 aborting False
>Thread-15626::WARNING::2013-03-18 07:58:29,611::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:58:29,614::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-22::DEBUG::2013-03-18 07:58:31,820::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:58:31,846::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764422 s, 5.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:58:33,397::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:58:33,424::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00103465 s, 4.0 MB/s\n'; <rc> = 0
>Thread-15664::DEBUG::2013-03-18 07:58:34,018::task::568::TaskManager.Task::(_updateState) Task=`460ec684-c739-4188-b9bb-406a504c0881`::moving from state init -> state preparing
>Thread-15664::INFO::2013-03-18 07:58:34,020::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15664::INFO::2013-03-18 07:58:34,021::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0286331176758', 'lastCheck': '0.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '4.4', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0281388759613', 'lastCheck': '2.2', 'code': 0, 'valid': True}}
>Thread-15664::DEBUG::2013-03-18 07:58:34,021::task::1151::TaskManager.Task::(prepare) Task=`460ec684-c739-4188-b9bb-406a504c0881`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0286331176758', 'lastCheck': '0.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '4.4', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0281388759613', 'lastCheck': '2.2', 'code': 0, 'valid': True}}
>Thread-15664::DEBUG::2013-03-18 07:58:34,021::task::568::TaskManager.Task::(_updateState) Task=`460ec684-c739-4188-b9bb-406a504c0881`::moving from state preparing -> state finished
>Thread-15664::DEBUG::2013-03-18 07:58:34,022::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15664::DEBUG::2013-03-18 07:58:34,022::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15664::DEBUG::2013-03-18 07:58:34,022::task::957::TaskManager.Task::(_decref) Task=`460ec684-c739-4188-b9bb-406a504c0881`::ref 0 aborting False
>Thread-21::DEBUG::2013-03-18 07:58:39,616::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 07:58:39,617::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 07:58:39,618::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 07:58:40,163::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 07:58:40,170::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 07:58:40,171::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:58:41,852::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 07:58:41,855::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 07:58:41,856::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:58:41,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.000828363 s, 2.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:58:41,885::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.000828363 s, 2.5 MB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 07:58:41,886::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 07:58:41,886::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 07:58:41,888::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 07:58:41,889::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:58:41,889::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:58:41,890::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:58:41,890::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:58:41,890::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:58:41,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:58:42,351::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:58:42,354::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 07:58:42,355::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::DEBUG::2013-03-18 07:58:42,357::lvm::339::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 07:58:42,357::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm pvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,size,vg_name,vg_uuid,pe_start,pe_count,pe_alloc_count,mda_count,dev_size' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:58:42,827::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:58:42,829::lvm::364::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::WARNING::2013-03-18 07:58:42,830::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 07:58:42,831::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 07:58:42,832::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 07:58:42,833::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:58:43,254::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 07:58:43,256::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:58:43,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00551771 s, 742 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:58:43,464::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:58:43,515::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.010312 s, 397 kB/s\n'; <rc> = 0
>Thread-15671::DEBUG::2013-03-18 07:58:44,330::task::568::TaskManager.Task::(_updateState) Task=`b03068ee-a70f-47ce-bc4c-c5d0f827d139`::moving from state init -> state preparing
>Thread-15671::INFO::2013-03-18 07:58:44,330::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15671::INFO::2013-03-18 07:58:44,331::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0535049438477', 'lastCheck': '0.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '14.7', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0494270324707', 'lastCheck': '1.0', 'code': 0, 'valid': True}}
>Thread-15671::DEBUG::2013-03-18 07:58:44,331::task::1151::TaskManager.Task::(prepare) Task=`b03068ee-a70f-47ce-bc4c-c5d0f827d139`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0535049438477', 'lastCheck': '0.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '14.7', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0494270324707', 'lastCheck': '1.0', 'code': 0, 'valid': True}}
>Thread-15671::DEBUG::2013-03-18 07:58:44,332::task::568::TaskManager.Task::(_updateState) Task=`b03068ee-a70f-47ce-bc4c-c5d0f827d139`::moving from state preparing -> state finished
>Thread-15671::DEBUG::2013-03-18 07:58:44,332::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15671::DEBUG::2013-03-18 07:58:44,332::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15671::DEBUG::2013-03-18 07:58:44,333::task::957::TaskManager.Task::(_decref) Task=`b03068ee-a70f-47ce-bc4c-c5d0f827d139`::ref 0 aborting False
>Thread-15675::DEBUG::2013-03-18 07:58:49,631::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15675::DEBUG::2013-03-18 07:58:49,631::task::568::TaskManager.Task::(_updateState) Task=`7b00746a-f1c9-4c2d-9025-f328098a85a5`::moving from state init -> state preparing
>Thread-15675::INFO::2013-03-18 07:58:49,632::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15675::INFO::2013-03-18 07:58:49,633::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}}
>Thread-15675::DEBUG::2013-03-18 07:58:49,633::task::1151::TaskManager.Task::(prepare) Task=`7b00746a-f1c9-4c2d-9025-f328098a85a5`::finished: {'spm_st': {'spmId': 1, 'spmStatus': 'Free', 'spmLver': 0}}
>Thread-15675::DEBUG::2013-03-18 07:58:49,633::task::568::TaskManager.Task::(_updateState) Task=`7b00746a-f1c9-4c2d-9025-f328098a85a5`::moving from state preparing -> state finished
>Thread-15675::DEBUG::2013-03-18 07:58:49,633::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15675::DEBUG::2013-03-18 07:58:49,634::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15675::DEBUG::2013-03-18 07:58:49,634::task::957::TaskManager.Task::(_decref) Task=`7b00746a-f1c9-4c2d-9025-f328098a85a5`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:58:53,309::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:58:53,342::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000857984 s, 4.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:58:53,560::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 07:58:53,563::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 07:58:53,564::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:58:53,596::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00122891 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:58:53,598::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00122891 s, 1.7 MB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 07:58:53,598::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 07:58:53,599::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 07:58:53,600::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 07:58:53,601::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:58:53,602::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:58:53,602::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:58:53,603::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:58:53,603::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:58:53,604::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:58:54,074::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:58:54,079::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:58:54,080::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'POOL_SPM_LVER=0', 'POOL_SPM_ID=-1', '_SHA_CKSUM=cc5065d8b2932b39a7a3c19e06d2678b4fa56441']
>Thread-20::WARNING::2013-03-18 07:58:54,082::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 07:58:54,084::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 07:58:54,084::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-20::DEBUG::2013-03-18 07:58:54,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:58:54,435::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:58:54,437::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:58:54,473::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00845564 s, 484 kB/s\n'; <rc> = 0
>Thread-15678::DEBUG::2013-03-18 07:58:54,567::task::568::TaskManager.Task::(_updateState) Task=`76c84959-c3d5-4440-86fe-df92480ed1e9`::moving from state init -> state preparing
>Thread-15678::INFO::2013-03-18 07:58:54,569::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15678::INFO::2013-03-18 07:58:54,569::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0378551483154', 'lastCheck': '0.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '25.0', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0355260372162', 'lastCheck': '1.2', 'code': 0, 'valid': True}}
>Thread-15678::DEBUG::2013-03-18 07:58:54,570::task::1151::TaskManager.Task::(prepare) Task=`76c84959-c3d5-4440-86fe-df92480ed1e9`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0378551483154', 'lastCheck': '0.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '25.0', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0355260372162', 'lastCheck': '1.2', 'code': 0, 'valid': True}}
>Thread-15678::DEBUG::2013-03-18 07:58:54,570::task::568::TaskManager.Task::(_updateState) Task=`76c84959-c3d5-4440-86fe-df92480ed1e9`::moving from state preparing -> state finished
>Thread-15678::DEBUG::2013-03-18 07:58:54,571::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15678::DEBUG::2013-03-18 07:58:54,571::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15678::DEBUG::2013-03-18 07:58:54,573::task::957::TaskManager.Task::(_decref) Task=`76c84959-c3d5-4440-86fe-df92480ed1e9`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:59:03,348::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:59:03,401::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.021657 s, 189 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:59:04,513::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:59:04,558::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000852296 s, 4.8 MB/s\n'; <rc> = 0
>Thread-15684::DEBUG::2013-03-18 07:59:04,853::task::568::TaskManager.Task::(_updateState) Task=`fac0c476-e560-4b08-9d4e-d23028662e0e`::moving from state init -> state preparing
>Thread-15684::INFO::2013-03-18 07:59:04,854::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15684::INFO::2013-03-18 07:59:04,855::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0470988750458', 'lastCheck': '0.3', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '35.2', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.054652929306', 'lastCheck': '1.4', 'code': 0, 'valid': True}}
>Thread-15684::DEBUG::2013-03-18 07:59:04,855::task::1151::TaskManager.Task::(prepare) Task=`fac0c476-e560-4b08-9d4e-d23028662e0e`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0470988750458', 'lastCheck': '0.3', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '35.2', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.054652929306', 'lastCheck': '1.4', 'code': 0, 'valid': True}}
>Thread-15684::DEBUG::2013-03-18 07:59:04,855::task::568::TaskManager.Task::(_updateState) Task=`fac0c476-e560-4b08-9d4e-d23028662e0e`::moving from state preparing -> state finished
>Thread-15684::DEBUG::2013-03-18 07:59:04,855::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15684::DEBUG::2013-03-18 07:59:04,856::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15684::DEBUG::2013-03-18 07:59:04,856::task::957::TaskManager.Task::(_decref) Task=`fac0c476-e560-4b08-9d4e-d23028662e0e`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:59:13,406::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:59:13,438::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.002404 s, 1.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:59:14,599::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:59:14,625::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00097622 s, 4.2 MB/s\n'; <rc> = 0
>Thread-15690::DEBUG::2013-03-18 07:59:15,163::task::568::TaskManager.Task::(_updateState) Task=`eb462d94-e24d-4554-9547-d6eb9e63790f`::moving from state init -> state preparing
>Thread-15690::INFO::2013-03-18 07:59:15,164::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15690::INFO::2013-03-18 07:59:15,165::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0280079841614', 'lastCheck': '0.5', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '45.5', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0342321395874', 'lastCheck': '1.7', 'code': 0, 'valid': True}}
>Thread-15690::DEBUG::2013-03-18 07:59:15,165::task::1151::TaskManager.Task::(prepare) Task=`eb462d94-e24d-4554-9547-d6eb9e63790f`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0280079841614', 'lastCheck': '0.5', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '45.5', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0342321395874', 'lastCheck': '1.7', 'code': 0, 'valid': True}}
>Thread-15690::DEBUG::2013-03-18 07:59:15,165::task::568::TaskManager.Task::(_updateState) Task=`eb462d94-e24d-4554-9547-d6eb9e63790f`::moving from state preparing -> state finished
>Thread-15690::DEBUG::2013-03-18 07:59:15,166::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15690::DEBUG::2013-03-18 07:59:15,166::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15690::DEBUG::2013-03-18 07:59:15,166::task::957::TaskManager.Task::(_decref) Task=`eb462d94-e24d-4554-9547-d6eb9e63790f`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:59:23,443::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:59:23,477::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00660374 s, 620 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:59:24,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:59:24,712::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00632118 s, 648 kB/s\n'; <rc> = 0
>Thread-15696::DEBUG::2013-03-18 07:59:25,381::task::568::TaskManager.Task::(_updateState) Task=`65758657-536c-4471-a9db-739923de18dc`::moving from state init -> state preparing
>Thread-15696::INFO::2013-03-18 07:59:25,382::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15696::INFO::2013-03-18 07:59:25,383::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0488250255585', 'lastCheck': '0.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '55.8', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0358901023865', 'lastCheck': '1.9', 'code': 0, 'valid': True}}
>Thread-15696::DEBUG::2013-03-18 07:59:25,383::task::1151::TaskManager.Task::(prepare) Task=`65758657-536c-4471-a9db-739923de18dc`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0488250255585', 'lastCheck': '0.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '55.8', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0358901023865', 'lastCheck': '1.9', 'code': 0, 'valid': True}}
>Thread-15696::DEBUG::2013-03-18 07:59:25,383::task::568::TaskManager.Task::(_updateState) Task=`65758657-536c-4471-a9db-739923de18dc`::moving from state preparing -> state finished
>Thread-15696::DEBUG::2013-03-18 07:59:25,384::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15696::DEBUG::2013-03-18 07:59:25,384::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15696::DEBUG::2013-03-18 07:59:25,384::task::957::TaskManager.Task::(_decref) Task=`65758657-536c-4471-a9db-739923de18dc`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:59:33,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:59:33,529::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00191593 s, 2.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:59:34,752::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:59:34,797::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00142258 s, 2.9 MB/s\n'; <rc> = 0
>Thread-15702::DEBUG::2013-03-18 07:59:35,658::task::568::TaskManager.Task::(_updateState) Task=`22954948-406d-44b3-947b-4775425f71f4`::moving from state init -> state preparing
>Thread-15702::INFO::2013-03-18 07:59:35,659::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15702::INFO::2013-03-18 07:59:35,660::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0471279621124', 'lastCheck': '0.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '66.0', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0466051101685', 'lastCheck': '2.1', 'code': 0, 'valid': True}}
>Thread-15702::DEBUG::2013-03-18 07:59:35,660::task::1151::TaskManager.Task::(prepare) Task=`22954948-406d-44b3-947b-4775425f71f4`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0471279621124', 'lastCheck': '0.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '66.0', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0466051101685', 'lastCheck': '2.1', 'code': 0, 'valid': True}}
>Thread-15702::DEBUG::2013-03-18 07:59:35,660::task::568::TaskManager.Task::(_updateState) Task=`22954948-406d-44b3-947b-4775425f71f4`::moving from state preparing -> state finished
>Thread-15702::DEBUG::2013-03-18 07:59:35,661::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15702::DEBUG::2013-03-18 07:59:35,661::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15702::DEBUG::2013-03-18 07:59:35,661::task::957::TaskManager.Task::(_decref) Task=`22954948-406d-44b3-947b-4775425f71f4`::ref 0 aborting False
>Thread-15668::WARNING::2013-03-18 07:59:40,197::fileSD::539::scanDomains::(collectMetaFiles) Metadata collection for domain path /rhev/data-center/mnt/10.34.1.147:_rhev_isos timedout
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/fileSD.py", line 528, in collectMetaFiles
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 275, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 180, in callCrabRPCFunction
>  File "/usr/share/vdsm/storage/remoteFileHandler.py", line 146, in _recvAll
>Timeout
>Thread-21::ERROR::2013-03-18 07:59:40,200::domainMonitor::208::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain 848fa27a-00f3-4bb4-b4da-101cd912b66e monitoring information
>Traceback (most recent call last):
>  File "/usr/share/vdsm/storage/domainMonitor.py", line 186, in _monitorDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__
>  File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain
>  File "/usr/share/vdsm/storage/sdc.py", line 121, in _realProduce
>  File "/usr/share/vdsm/storage/sdc.py", line 152, in _findDomain
>StorageDomainDoesNotExist: Storage domain does not exist: (u'848fa27a-00f3-4bb4-b4da-101cd912b66e',)
>Thread-15707::DEBUG::2013-03-18 07:59:42,982::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15707::DEBUG::2013-03-18 07:59:42,983::task::568::TaskManager.Task::(_updateState) Task=`b73b8c89-6250-482f-822a-45042a520f69`::moving from state init -> state preparing
>Thread-15707::INFO::2013-03-18 07:59:42,983::logUtils::37::dispatcher::(wrapper) Run and protect: spmStart(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', prevID=1, prevLVER='0', recoveryMode=None, scsiFencing='true', maxHostID=250, domVersion='3', options=None)
>Thread-15707::DEBUG::2013-03-18 07:59:42,984::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`2dcd63ea-f6bd-487b-addc-e2d115ee559d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15707::DEBUG::2013-03-18 07:59:42,985::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'exclusive'
>Thread-15707::DEBUG::2013-03-18 07:59:42,985::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'exclusive' (1 active user)
>Thread-15707::DEBUG::2013-03-18 07:59:42,986::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`2dcd63ea-f6bd-487b-addc-e2d115ee559d`::Granted request
>Thread-15707::DEBUG::2013-03-18 07:59:42,986::task::794::TaskManager.Task::(resourceAcquired) Task=`b73b8c89-6250-482f-822a-45042a520f69`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (exclusive)
>Thread-15707::DEBUG::2013-03-18 07:59:42,987::task::957::TaskManager.Task::(_decref) Task=`b73b8c89-6250-482f-822a-45042a520f69`::ref 1 aborting False
>Thread-15707::INFO::2013-03-18 07:59:42,987::logUtils::39::dispatcher::(wrapper) Run and protect: spmStart, Return response: None
>Thread-15707::DEBUG::2013-03-18 07:59:42,987::task::1146::TaskManager.Task::(prepare) Task=`b73b8c89-6250-482f-822a-45042a520f69`::Prepare: 1 jobs exist, move to acquiring
>Thread-15707::DEBUG::2013-03-18 07:59:42,988::task::568::TaskManager.Task::(_updateState) Task=`b73b8c89-6250-482f-822a-45042a520f69`::moving from state preparing -> state acquiring
>Thread-15707::DEBUG::2013-03-18 07:59:42,988::task::568::TaskManager.Task::(_updateState) Task=`b73b8c89-6250-482f-822a-45042a520f69`::moving from state acquiring -> state queued
>Thread-15707::DEBUG::2013-03-18 07:59:42,988::taskManager::50::TaskManager::(_queueTask) queuing task: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15707::DEBUG::2013-03-18 07:59:43,000::taskManager::56::TaskManager::(_queueTask) task queued: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15707::DEBUG::2013-03-18 07:59:43,001::task::1148::TaskManager.Task::(prepare) Task=`b73b8c89-6250-482f-822a-45042a520f69`::returning
>Thread-15707::DEBUG::2013-03-18 07:59:43,001::task::957::TaskManager.Task::(_decref) Task=`b73b8c89-6250-482f-822a-45042a520f69`::ref 0 aborting False
>Thread-2::DEBUG::2013-03-18 07:59:43,003::threadPool::55::Misc.ThreadPool::(setRunningTask) Number of running tasks: 1
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:43,004::threadPool::200::Misc.ThreadPool.WorkerThread::(run) Task: b73b8c89-6250-482f-822a-45042a520f69 running: <bound method Task.commit of <storage.task.Task instance at 0x1279908>> with: None
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:43,004::task::1159::TaskManager.Task::(commit) Task=`b73b8c89-6250-482f-822a-45042a520f69`::committing task: b73b8c89-6250-482f-822a-45042a520f69
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:43,005::task::568::TaskManager.Task::(_updateState) Task=`b73b8c89-6250-482f-822a-45042a520f69`::moving from state queued -> state running
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:43,005::task::872::TaskManager.Task::(_runJobs) Task=`b73b8c89-6250-482f-822a-45042a520f69`::Task.run: running job 0: spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x129db50>> (args: (1, '0', 'true', 250, 3) kwargs: {})
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:43,005::task::304::TaskManager.Task::(run) Task=`b73b8c89-6250-482f-822a-45042a520f69`::Job.run: running spmStart: <bound method StoragePool.startSpm of <storage.sp.StoragePool object at 0x129db50>> (args: (1, '0', 'true', 250, 3) kwargs: {}) callback None
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:43,006::misc::1025::SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads)
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:43,006::misc::1027::SamplingMethod::(__call__) Got in to sampling method
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:43,007::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:43,008::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:43,008::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:43,009::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:43,009::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:43,010::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:59:43,540::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:59:43,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0329595 s, 124 kB/s\n'; <rc> = 0
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:43,767::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:43,771::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:43,772::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'POOL_SPM_LVER=0', 'POOL_SPM_ID=-1', '_SHA_CKSUM=cc5065d8b2932b39a7a3c19e06d2678b4fa56441']
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:43,773::misc::1035::SamplingMethod::(__call__) Returning last result
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:43,773::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:43,774::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:43,774::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:43,774::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:43,775::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:43,776::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-15709::DEBUG::2013-03-18 07:59:44,034::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15709::DEBUG::2013-03-18 07:59:44,036::task::568::TaskManager.Task::(_updateState) Task=`86a8ec7b-869b-48dd-a344-a6c8964a257c`::moving from state init -> state preparing
>Thread-15709::INFO::2013-03-18 07:59:44,037::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15709::DEBUG::2013-03-18 07:59:44,038::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15709::DEBUG::2013-03-18 07:59:44,038::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15709::INFO::2013-03-18 07:59:44,039::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15709::DEBUG::2013-03-18 07:59:44,039::task::1151::TaskManager.Task::(prepare) Task=`86a8ec7b-869b-48dd-a344-a6c8964a257c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15709::DEBUG::2013-03-18 07:59:44,039::task::568::TaskManager.Task::(_updateState) Task=`86a8ec7b-869b-48dd-a344-a6c8964a257c`::moving from state preparing -> state finished
>Thread-15709::DEBUG::2013-03-18 07:59:44,039::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15709::DEBUG::2013-03-18 07:59:44,040::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15709::DEBUG::2013-03-18 07:59:44,040::task::957::TaskManager.Task::(_decref) Task=`86a8ec7b-869b-48dd-a344-a6c8964a257c`::ref 0 aborting False
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,138::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,141::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,142::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'POOL_SPM_LVER=0', 'POOL_SPM_ID=-1', '_SHA_CKSUM=cc5065d8b2932b39a7a3c19e06d2678b4fa56441']
>b73b8c89-6250-482f-822a-45042a520f69::INFO::2013-03-18 07:59:44,143::sp::247::Storage.StoragePool::(startSpm) expected previd:-1 lver:0 got request for previd:1 lver:0
>b73b8c89-6250-482f-822a-45042a520f69::INFO::2013-03-18 07:59:44,143::clusterlock::172::SANLock::(acquireHostId) Acquiring host id for domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a (id: 2)
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,145::clusterlock::190::SANLock::(acquireHostId) Host id for domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a successfully acquired (id: 2)
>b73b8c89-6250-482f-822a-45042a520f69::WARNING::2013-03-18 07:59:44,146::fileUtils::184::fileUtils::(createdir) Dir /rhev/data-center/mnt/blockSD/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/images already exists
>b73b8c89-6250-482f-822a-45042a520f69::WARNING::2013-03-18 07:59:44,147::fileUtils::184::fileUtils::(createdir) Dir /rhev/data-center/mnt/blockSD/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/dom_md already exists
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,148::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,148::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,149::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,149::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,149::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,150::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,181::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00672171 s, 305 kB/s\n'; <rc> = 0
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,183::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00672171 s, 305 kB/s'], size: 2048
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,184::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>b73b8c89-6250-482f-822a-45042a520f69::WARNING::2013-03-18 07:59:44,184::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,187::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,188::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,189::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,189::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,190::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,190::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,192::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,642::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,647::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,648::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'POOL_SPM_LVER=0', 'POOL_SPM_ID=-1', '_SHA_CKSUM=cc5065d8b2932b39a7a3c19e06d2678b4fa56441']
>b73b8c89-6250-482f-822a-45042a520f69::INFO::2013-03-18 07:59:44,649::clusterlock::223::SANLock::(acquire) Acquiring cluster lock for domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a (id: 2)
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,735::clusterlock::247::SANLock::(acquire) Cluster lock for domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a successfully acquired (id: 2)
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,736::sp::259::Storage.StoragePool::(startSpm) spm lock acquired successfully
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,737::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,737::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,737::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,738::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,738::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:44,739::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:59:44,842::lvm::373::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-20::DEBUG::2013-03-18 07:59:44,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-15710::DEBUG::2013-03-18 07:59:45,080::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15710::DEBUG::2013-03-18 07:59:45,083::task::568::TaskManager.Task::(_updateState) Task=`2e89e70d-3230-447c-8f68-4e0f5375871f`::moving from state init -> state preparing
>Thread-15710::INFO::2013-03-18 07:59:45,084::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15710::DEBUG::2013-03-18 07:59:45,085::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15710::DEBUG::2013-03-18 07:59:45,085::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15710::INFO::2013-03-18 07:59:45,085::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15710::DEBUG::2013-03-18 07:59:45,086::task::1151::TaskManager.Task::(prepare) Task=`2e89e70d-3230-447c-8f68-4e0f5375871f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15710::DEBUG::2013-03-18 07:59:45,086::task::568::TaskManager.Task::(_updateState) Task=`2e89e70d-3230-447c-8f68-4e0f5375871f`::moving from state preparing -> state finished
>Thread-15710::DEBUG::2013-03-18 07:59:45,086::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15710::DEBUG::2013-03-18 07:59:45,086::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15710::DEBUG::2013-03-18 07:59:45,087::task::957::TaskManager.Task::(_decref) Task=`2e89e70d-3230-447c-8f68-4e0f5375871f`::ref 0 aborting False
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:45,232::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:45,236::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'POOL_SPM_LVER=0', 'POOL_SPM_ID=-1', '_SHA_CKSUM=cc5065d8b2932b39a7a3c19e06d2678b4fa56441']
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:45,237::persistentDict::167::Storage.PersistentDict::(transaction) Starting transaction
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:45,238::persistentDict::173::Storage.PersistentDict::(transaction) Flushing changes
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:45,238::persistentDict::299::Storage.PersistentDict::(flush) about to write lines (VGTagMetadataRW)=['CLASS=Data', 'DESCRIPTION=msd', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'LOGBLKSIZE=512', 'MASTER_VERSION=1', 'PHYBLKSIZE=512', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=1', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'ROLE=Master', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'TYPE=ISCSI', 'VERSION=3', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', '_SHA_CKSUM=9b6c31302edce2645b9361d66bf5bc8091db2bcb']
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:45,239::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm reload operation' is holding the operation mutex, waiting...
>Thread-20::DEBUG::2013-03-18 07:59:45,256::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:59:45,258::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:59:45,262::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:45,263::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:45,302::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:45,310::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 07:59:45,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00833856 s, 491 kB/s\n'; <rc> = 0
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:45,315::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 07:59:45,319::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:45,323::lvm::373::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-20::DEBUG::2013-03-18 07:59:45,326::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:45,349::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:59:46,013::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-15711::DEBUG::2013-03-18 07:59:46,030::task::568::TaskManager.Task::(_updateState) Task=`5d614afa-910c-44e8-afdd-dfab68d22622`::moving from state init -> state preparing
>Thread-15711::INFO::2013-03-18 07:59:46,031::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15711::INFO::2013-03-18 07:59:46,032::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0471279621124', 'lastCheck': '11.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '5.8', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.103109121323', 'lastCheck': '2.4', 'code': 0, 'valid': True}}
>Thread-15711::DEBUG::2013-03-18 07:59:46,033::task::1151::TaskManager.Task::(prepare) Task=`5d614afa-910c-44e8-afdd-dfab68d22622`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0471279621124', 'lastCheck': '11.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '5.8', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.103109121323', 'lastCheck': '2.4', 'code': 0, 'valid': True}}
>Thread-15711::DEBUG::2013-03-18 07:59:46,034::task::568::TaskManager.Task::(_updateState) Task=`5d614afa-910c-44e8-afdd-dfab68d22622`::moving from state preparing -> state finished
>Thread-15711::DEBUG::2013-03-18 07:59:46,034::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15711::DEBUG::2013-03-18 07:59:46,034::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15711::DEBUG::2013-03-18 07:59:46,035::task::957::TaskManager.Task::(_decref) Task=`5d614afa-910c-44e8-afdd-dfab68d22622`::ref 0 aborting False
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:46,038::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:46,040::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:46,041::blockSD::297::storage.Metadata.VGTagMetadataRW::(writelines) Updating metadata adding=MDT_POOL_SPM_LVER=1, MDT__SHA_CKSUM=9b6c31302edce2645b9361d66bf5bc8091db2bcb, MDT_POOL_SPM_ID=2 removing=MDT_POOL_SPM_ID=-1, MDT__SHA_CKSUM=cc5065d8b2932b39a7a3c19e06d2678b4fa56441, MDT_POOL_SPM_LVER=0
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:46,042::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgchange --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --deltag MDT_POOL_SPM_ID=-1 --deltag MDT__SHA_CKSUM=cc5065d8b2932b39a7a3c19e06d2678b4fa56441 --deltag MDT_POOL_SPM_LVER=0 --addtag MDT_POOL_SPM_ID=2 --addtag MDT__SHA_CKSUM=9b6c31302edce2645b9361d66bf5bc8091db2bcb --addtag MDT_POOL_SPM_LVER=1 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-15712::DEBUG::2013-03-18 07:59:46,112::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15712::DEBUG::2013-03-18 07:59:46,114::task::568::TaskManager.Task::(_updateState) Task=`9ccbd3b2-6da1-46ce-81ec-9e55e65b54c6`::moving from state init -> state preparing
>Thread-15712::INFO::2013-03-18 07:59:46,115::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15712::DEBUG::2013-03-18 07:59:46,123::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15712::DEBUG::2013-03-18 07:59:46,126::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15712::INFO::2013-03-18 07:59:46,126::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15712::DEBUG::2013-03-18 07:59:46,126::task::1151::TaskManager.Task::(prepare) Task=`9ccbd3b2-6da1-46ce-81ec-9e55e65b54c6`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15712::DEBUG::2013-03-18 07:59:46,126::task::568::TaskManager.Task::(_updateState) Task=`9ccbd3b2-6da1-46ce-81ec-9e55e65b54c6`::moving from state preparing -> state finished
>Thread-15712::DEBUG::2013-03-18 07:59:46,127::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15712::DEBUG::2013-03-18 07:59:46,127::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15712::DEBUG::2013-03-18 07:59:46,127::task::957::TaskManager.Task::(_decref) Task=`9ccbd3b2-6da1-46ce-81ec-9e55e65b54c6`::ref 0 aborting False
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:46,552::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:46,553::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:46,554::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:46,554::persistentDict::175::Storage.PersistentDict::(transaction) Finished transaction
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:46,565::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`d266ca2f-281c-4aea-b897-8b93e1b31548`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:46,566::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'exclusive'
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:46,570::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.upgrade_a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'exclusive' (1 active user)
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:46,571::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.upgrade_a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`d266ca2f-281c-4aea-b897-8b93e1b31548`::Granted request
>b73b8c89-6250-482f-822a-45042a520f69::INFO::2013-03-18 07:59:46,576::sp::408::Storage.StoragePool::(_upgradePool) Trying to upgrade master domain `4bf968ef-020a-47ee-a1ae-f85eaeb8751a`
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:46,578::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.4bf968ef-020a-47ee-a1ae-f85eaeb8751a`ReqID=`b7104b68-33aa-41ce-a307-66fdaf946424`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:46,579::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.4bf968ef-020a-47ee-a1ae-f85eaeb8751a' for lock type 'exclusive'
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:46,581::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.4bf968ef-020a-47ee-a1ae-f85eaeb8751a' is free. Now locking as 'exclusive' (1 active user)
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:46,582::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.4bf968ef-020a-47ee-a1ae-f85eaeb8751a`ReqID=`b7104b68-33aa-41ce-a307-66fdaf946424`::Granted request
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:46,588::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:46,594::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,045::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,045::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,046::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,087::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.0112579 s, 182 kB/s\n'; <rc> = 0
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,089::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.0112579 s, 182 kB/s'], size: 2048
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,090::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>b73b8c89-6250-482f-822a-45042a520f69::WARNING::2013-03-18 07:59:47,090::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,093::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,094::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,094::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,095::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,095::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,096::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,097::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-15714::DEBUG::2013-03-18 07:59:47,161::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15714::DEBUG::2013-03-18 07:59:47,164::task::568::TaskManager.Task::(_updateState) Task=`4dd14a2a-2089-4958-883e-9086e3dc10ad`::moving from state init -> state preparing
>Thread-15714::INFO::2013-03-18 07:59:47,165::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15714::DEBUG::2013-03-18 07:59:47,166::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15714::DEBUG::2013-03-18 07:59:47,166::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15714::INFO::2013-03-18 07:59:47,166::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15714::DEBUG::2013-03-18 07:59:47,167::task::1151::TaskManager.Task::(prepare) Task=`4dd14a2a-2089-4958-883e-9086e3dc10ad`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15714::DEBUG::2013-03-18 07:59:47,167::task::568::TaskManager.Task::(_updateState) Task=`4dd14a2a-2089-4958-883e-9086e3dc10ad`::moving from state preparing -> state finished
>Thread-15714::DEBUG::2013-03-18 07:59:47,167::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15714::DEBUG::2013-03-18 07:59:47,167::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15714::DEBUG::2013-03-18 07:59:47,168::task::957::TaskManager.Task::(_decref) Task=`4dd14a2a-2089-4958-883e-9086e3dc10ad`::ref 0 aborting False
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,541::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,542::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'POOL_SPM_ID=2', '_SHA_CKSUM=9b6c31302edce2645b9361d66bf5bc8091db2bcb', 'POOL_SPM_LVER=1']
>b73b8c89-6250-482f-822a-45042a520f69::WARNING::2013-03-18 07:59:47,544::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>b73b8c89-6250-482f-822a-45042a520f69::WARNING::2013-03-18 07:59:47,545::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>b73b8c89-6250-482f-822a-45042a520f69::WARNING::2013-03-18 07:59:47,546::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,551::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.4bf968ef-020a-47ee-a1ae-f85eaeb8751a'
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,551::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (0 active users)
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,552::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.4bf968ef-020a-47ee-a1ae-f85eaeb8751a' is free, finding out if anyone is waiting for it.
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,552::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.4bf968ef-020a-47ee-a1ae-f85eaeb8751a', Clearing records.
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,552::sp::412::Storage.StoragePool::(_upgradePool) Marking all domains for upgrade
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,553::sp::419::Storage.StoragePool::(_upgradePool) Registering with state change event
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,553::sp::421::Storage.StoragePool::(_upgradePool) Running initial domain upgrade threads
>Thread-15715::DEBUG::2013-03-18 07:59:47,555::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-15715::DEBUG::2013-03-18 07:59:47,556::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,581::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15716::DEBUG::2013-03-18 07:59:47,588::sp::167::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-15716::DEBUG::2013-03-18 07:59:47,603::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_f33e50cc-eb0a-4720-857d-bc7001a72567`ReqID=`2e518b57-073c-44cb-aadf-88e430151056`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15716::DEBUG::2013-03-18 07:59:47,603::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_f33e50cc-eb0a-4720-857d-bc7001a72567' for lock type 'exclusive'
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,589::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,605::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.upgrade_a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,605::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,608::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/fsck -p /dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/master' (cwd None)
>Thread-15716::DEBUG::2013-03-18 07:59:47,643::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.upgrade_f33e50cc-eb0a-4720-857d-bc7001a72567' is free. Now locking as 'exclusive' (1 active user)
>Thread-15716::DEBUG::2013-03-18 07:59:47,646::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.upgrade_f33e50cc-eb0a-4720-857d-bc7001a72567`ReqID=`2e518b57-073c-44cb-aadf-88e430151056`::Granted request
>Thread-15716::DEBUG::2013-03-18 07:59:47,656::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.f33e50cc-eb0a-4720-857d-bc7001a72567`ReqID=`fada5047-2759-4105-b793-7ac775c008c8`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15716::DEBUG::2013-03-18 07:59:47,657::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.f33e50cc-eb0a-4720-857d-bc7001a72567' for lock type 'exclusive'
>Thread-15716::DEBUG::2013-03-18 07:59:47,657::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.f33e50cc-eb0a-4720-857d-bc7001a72567' is free. Now locking as 'exclusive' (1 active user)
>Thread-15716::DEBUG::2013-03-18 07:59:47,658::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.f33e50cc-eb0a-4720-857d-bc7001a72567`ReqID=`fada5047-2759-4105-b793-7ac775c008c8`::Granted request
>Thread-15716::DEBUG::2013-03-18 07:59:47,666::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm reload operation' is holding the operation mutex, waiting...
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,843::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,844::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/tune2fs -j /dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/master' (cwd None)
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,919::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = 'The filesystem already has a journal.\n'; <rc> = 1
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:47,921::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/mount -t ext3 /dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/master /rhev/data-center/mnt/blockSD/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/master' (cwd None)
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:48,029::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /bin/chown vdsm:kvm /rhev/data-center/mnt/blockSD/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/master' (cwd None)
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 07:59:48,114::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-15717::DEBUG::2013-03-18 07:59:48,200::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15717::DEBUG::2013-03-18 07:59:48,202::task::568::TaskManager.Task::(_updateState) Task=`c6afe879-befa-4485-95f4-31192f764389`::moving from state init -> state preparing
>Thread-15717::INFO::2013-03-18 07:59:48,203::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15717::DEBUG::2013-03-18 07:59:48,206::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15717::DEBUG::2013-03-18 07:59:48,207::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15717::INFO::2013-03-18 07:59:48,208::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15717::DEBUG::2013-03-18 07:59:48,208::task::1151::TaskManager.Task::(prepare) Task=`c6afe879-befa-4485-95f4-31192f764389`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15717::DEBUG::2013-03-18 07:59:48,208::task::568::TaskManager.Task::(_updateState) Task=`c6afe879-befa-4485-95f4-31192f764389`::moving from state preparing -> state finished
>Thread-15717::DEBUG::2013-03-18 07:59:48,209::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15717::DEBUG::2013-03-18 07:59:48,209::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15717::DEBUG::2013-03-18 07:59:48,210::task::957::TaskManager.Task::(_decref) Task=`c6afe879-befa-4485-95f4-31192f764389`::ref 0 aborting False
>Thread-15715::DEBUG::2013-03-18 07:59:48,304::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-15715::WARNING::2013-03-18 07:59:48,311::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-15715::DEBUG::2013-03-18 07:59:48,312::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-15716::DEBUG::2013-03-18 07:59:48,312::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-15716::DEBUG::2013-03-18 07:59:48,313::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-15716::DEBUG::2013-03-18 07:59:48,313::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-15716::DEBUG::2013-03-18 07:59:48,314::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-15716::DEBUG::2013-03-18 07:59:48,314::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-15716::DEBUG::2013-03-18 07:59:48,315::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-15716::DEBUG::2013-03-18 07:59:48,751::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-15716::DEBUG::2013-03-18 07:59:48,756::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-15716::DEBUG::2013-03-18 07:59:48,757::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-15716::DEBUG::2013-03-18 07:59:48,758::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.f33e50cc-eb0a-4720-857d-bc7001a72567'
>Thread-15716::DEBUG::2013-03-18 07:59:48,759::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.f33e50cc-eb0a-4720-857d-bc7001a72567' (0 active users)
>Thread-15716::DEBUG::2013-03-18 07:59:48,759::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.f33e50cc-eb0a-4720-857d-bc7001a72567' is free, finding out if anyone is waiting for it.
>Thread-15716::DEBUG::2013-03-18 07:59:48,759::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.f33e50cc-eb0a-4720-857d-bc7001a72567', Clearing records.
>Thread-15716::DEBUG::2013-03-18 07:59:48,760::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_f33e50cc-eb0a-4720-857d-bc7001a72567'
>Thread-15716::DEBUG::2013-03-18 07:59:48,760::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_f33e50cc-eb0a-4720-857d-bc7001a72567' (0 active users)
>Thread-15716::DEBUG::2013-03-18 07:59:48,760::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.upgrade_f33e50cc-eb0a-4720-857d-bc7001a72567' is free, finding out if anyone is waiting for it.
>Thread-15716::DEBUG::2013-03-18 07:59:48,761::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_f33e50cc-eb0a-4720-857d-bc7001a72567', Clearing records.
>Thread-15720::DEBUG::2013-03-18 07:59:49,266::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15720::DEBUG::2013-03-18 07:59:49,271::task::568::TaskManager.Task::(_updateState) Task=`f55ffeda-86fa-4856-9230-9044e8567466`::moving from state init -> state preparing
>Thread-15720::INFO::2013-03-18 07:59:49,272::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15720::DEBUG::2013-03-18 07:59:49,273::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15720::DEBUG::2013-03-18 07:59:49,273::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15720::INFO::2013-03-18 07:59:49,273::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15720::DEBUG::2013-03-18 07:59:49,273::task::1151::TaskManager.Task::(prepare) Task=`f55ffeda-86fa-4856-9230-9044e8567466`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15720::DEBUG::2013-03-18 07:59:49,273::task::568::TaskManager.Task::(_updateState) Task=`f55ffeda-86fa-4856-9230-9044e8567466`::moving from state preparing -> state finished
>Thread-15720::DEBUG::2013-03-18 07:59:49,274::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15720::DEBUG::2013-03-18 07:59:49,274::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15720::DEBUG::2013-03-18 07:59:49,274::task::957::TaskManager.Task::(_decref) Task=`f55ffeda-86fa-4856-9230-9044e8567466`::ref 0 aborting False
>Thread-15722::DEBUG::2013-03-18 07:59:50,297::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15722::DEBUG::2013-03-18 07:59:50,298::task::568::TaskManager.Task::(_updateState) Task=`445fac9a-76a3-4ae9-a8ed-f998dfe9f1ae`::moving from state init -> state preparing
>Thread-15722::INFO::2013-03-18 07:59:50,298::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15722::DEBUG::2013-03-18 07:59:50,298::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15722::DEBUG::2013-03-18 07:59:50,299::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15722::INFO::2013-03-18 07:59:50,299::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15722::DEBUG::2013-03-18 07:59:50,299::task::1151::TaskManager.Task::(prepare) Task=`445fac9a-76a3-4ae9-a8ed-f998dfe9f1ae`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15722::DEBUG::2013-03-18 07:59:50,299::task::568::TaskManager.Task::(_updateState) Task=`445fac9a-76a3-4ae9-a8ed-f998dfe9f1ae`::moving from state preparing -> state finished
>Thread-15722::DEBUG::2013-03-18 07:59:50,300::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15722::DEBUG::2013-03-18 07:59:50,300::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15722::DEBUG::2013-03-18 07:59:50,300::task::957::TaskManager.Task::(_decref) Task=`445fac9a-76a3-4ae9-a8ed-f998dfe9f1ae`::ref 0 aborting False
>Thread-15723::DEBUG::2013-03-18 07:59:51,325::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15723::DEBUG::2013-03-18 07:59:51,326::task::568::TaskManager.Task::(_updateState) Task=`89a19c37-7a60-4b6a-b321-87fa9c4e69fc`::moving from state init -> state preparing
>Thread-15723::INFO::2013-03-18 07:59:51,326::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15723::DEBUG::2013-03-18 07:59:51,326::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15723::DEBUG::2013-03-18 07:59:51,326::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15723::INFO::2013-03-18 07:59:51,327::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15723::DEBUG::2013-03-18 07:59:51,327::task::1151::TaskManager.Task::(prepare) Task=`89a19c37-7a60-4b6a-b321-87fa9c4e69fc`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15723::DEBUG::2013-03-18 07:59:51,327::task::568::TaskManager.Task::(_updateState) Task=`89a19c37-7a60-4b6a-b321-87fa9c4e69fc`::moving from state preparing -> state finished
>Thread-15723::DEBUG::2013-03-18 07:59:51,327::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15723::DEBUG::2013-03-18 07:59:51,328::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15723::DEBUG::2013-03-18 07:59:51,328::task::957::TaskManager.Task::(_decref) Task=`89a19c37-7a60-4b6a-b321-87fa9c4e69fc`::ref 0 aborting False
>Thread-15725::DEBUG::2013-03-18 07:59:52,350::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15725::DEBUG::2013-03-18 07:59:52,351::task::568::TaskManager.Task::(_updateState) Task=`978026bf-a42c-4275-be71-459fcc867f8e`::moving from state init -> state preparing
>Thread-15725::INFO::2013-03-18 07:59:52,351::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15725::DEBUG::2013-03-18 07:59:52,351::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15725::DEBUG::2013-03-18 07:59:52,352::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15725::INFO::2013-03-18 07:59:52,352::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15725::DEBUG::2013-03-18 07:59:52,352::task::1151::TaskManager.Task::(prepare) Task=`978026bf-a42c-4275-be71-459fcc867f8e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15725::DEBUG::2013-03-18 07:59:52,352::task::568::TaskManager.Task::(_updateState) Task=`978026bf-a42c-4275-be71-459fcc867f8e`::moving from state preparing -> state finished
>Thread-15725::DEBUG::2013-03-18 07:59:52,353::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15725::DEBUG::2013-03-18 07:59:52,353::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15725::DEBUG::2013-03-18 07:59:52,353::task::957::TaskManager.Task::(_decref) Task=`978026bf-a42c-4275-be71-459fcc867f8e`::ref 0 aborting False
>Thread-15726::DEBUG::2013-03-18 07:59:53,373::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15726::DEBUG::2013-03-18 07:59:53,374::task::568::TaskManager.Task::(_updateState) Task=`4f10d753-4bdb-4abf-95b7-46384a421305`::moving from state init -> state preparing
>Thread-15726::INFO::2013-03-18 07:59:53,377::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15726::DEBUG::2013-03-18 07:59:53,377::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15726::DEBUG::2013-03-18 07:59:53,377::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15726::INFO::2013-03-18 07:59:53,377::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15726::DEBUG::2013-03-18 07:59:53,378::task::1151::TaskManager.Task::(prepare) Task=`4f10d753-4bdb-4abf-95b7-46384a421305`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15726::DEBUG::2013-03-18 07:59:53,378::task::568::TaskManager.Task::(_updateState) Task=`4f10d753-4bdb-4abf-95b7-46384a421305`::moving from state preparing -> state finished
>Thread-15726::DEBUG::2013-03-18 07:59:53,378::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15726::DEBUG::2013-03-18 07:59:53,378::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15726::DEBUG::2013-03-18 07:59:53,379::task::957::TaskManager.Task::(_decref) Task=`4f10d753-4bdb-4abf-95b7-46384a421305`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 07:59:53,665::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 07:59:53,706::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135862 s, 301 kB/s\n'; <rc> = 0
>Thread-15728::DEBUG::2013-03-18 07:59:54,402::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15728::DEBUG::2013-03-18 07:59:54,404::task::568::TaskManager.Task::(_updateState) Task=`561dbcb6-e905-4ab1-b6cc-0fb881f35379`::moving from state init -> state preparing
>Thread-15728::INFO::2013-03-18 07:59:54,405::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15728::DEBUG::2013-03-18 07:59:54,406::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15728::DEBUG::2013-03-18 07:59:54,406::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15728::INFO::2013-03-18 07:59:54,407::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15728::DEBUG::2013-03-18 07:59:54,407::task::1151::TaskManager.Task::(prepare) Task=`561dbcb6-e905-4ab1-b6cc-0fb881f35379`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15728::DEBUG::2013-03-18 07:59:54,407::task::568::TaskManager.Task::(_updateState) Task=`561dbcb6-e905-4ab1-b6cc-0fb881f35379`::moving from state preparing -> state finished
>Thread-15728::DEBUG::2013-03-18 07:59:54,408::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15728::DEBUG::2013-03-18 07:59:54,408::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15728::DEBUG::2013-03-18 07:59:54,408::task::957::TaskManager.Task::(_decref) Task=`561dbcb6-e905-4ab1-b6cc-0fb881f35379`::ref 0 aborting False
>Thread-15729::DEBUG::2013-03-18 07:59:55,432::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15729::DEBUG::2013-03-18 07:59:55,432::task::568::TaskManager.Task::(_updateState) Task=`6f841550-25ee-4e27-ac3c-95d6bd2d8b1c`::moving from state init -> state preparing
>Thread-15729::INFO::2013-03-18 07:59:55,433::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15729::DEBUG::2013-03-18 07:59:55,433::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15729::DEBUG::2013-03-18 07:59:55,433::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15729::INFO::2013-03-18 07:59:55,433::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15729::DEBUG::2013-03-18 07:59:55,434::task::1151::TaskManager.Task::(prepare) Task=`6f841550-25ee-4e27-ac3c-95d6bd2d8b1c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15729::DEBUG::2013-03-18 07:59:55,434::task::568::TaskManager.Task::(_updateState) Task=`6f841550-25ee-4e27-ac3c-95d6bd2d8b1c`::moving from state preparing -> state finished
>Thread-15729::DEBUG::2013-03-18 07:59:55,434::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15729::DEBUG::2013-03-18 07:59:55,434::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15729::DEBUG::2013-03-18 07:59:55,434::task::957::TaskManager.Task::(_decref) Task=`6f841550-25ee-4e27-ac3c-95d6bd2d8b1c`::ref 0 aborting False
>Thread-15730::DEBUG::2013-03-18 07:59:56,346::task::568::TaskManager.Task::(_updateState) Task=`f2ffd479-80a5-43e4-a5b7-cad6b5ed77ec`::moving from state init -> state preparing
>Thread-15730::INFO::2013-03-18 07:59:56,346::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15730::INFO::2013-03-18 07:59:56,347::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0559720993042', 'lastCheck': '8.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '16.1', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0436019897461', 'lastCheck': '2.6', 'code': 0, 'valid': True}}
>Thread-15730::DEBUG::2013-03-18 07:59:56,347::task::1151::TaskManager.Task::(prepare) Task=`f2ffd479-80a5-43e4-a5b7-cad6b5ed77ec`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0559720993042', 'lastCheck': '8.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '16.1', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0436019897461', 'lastCheck': '2.6', 'code': 0, 'valid': True}}
>Thread-15730::DEBUG::2013-03-18 07:59:56,347::task::568::TaskManager.Task::(_updateState) Task=`f2ffd479-80a5-43e4-a5b7-cad6b5ed77ec`::moving from state preparing -> state finished
>Thread-15730::DEBUG::2013-03-18 07:59:56,348::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15730::DEBUG::2013-03-18 07:59:56,348::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15730::DEBUG::2013-03-18 07:59:56,348::task::957::TaskManager.Task::(_decref) Task=`f2ffd479-80a5-43e4-a5b7-cad6b5ed77ec`::ref 0 aborting False
>Thread-15732::DEBUG::2013-03-18 07:59:56,471::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15732::DEBUG::2013-03-18 07:59:56,471::task::568::TaskManager.Task::(_updateState) Task=`18457ee7-f9ae-49fc-b7d1-040ca0e44d12`::moving from state init -> state preparing
>Thread-15732::INFO::2013-03-18 07:59:56,472::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15732::DEBUG::2013-03-18 07:59:56,472::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15732::DEBUG::2013-03-18 07:59:56,472::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15732::INFO::2013-03-18 07:59:56,472::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15732::DEBUG::2013-03-18 07:59:56,473::task::1151::TaskManager.Task::(prepare) Task=`18457ee7-f9ae-49fc-b7d1-040ca0e44d12`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15732::DEBUG::2013-03-18 07:59:56,473::task::568::TaskManager.Task::(_updateState) Task=`18457ee7-f9ae-49fc-b7d1-040ca0e44d12`::moving from state preparing -> state finished
>Thread-15732::DEBUG::2013-03-18 07:59:56,473::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15732::DEBUG::2013-03-18 07:59:56,473::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15732::DEBUG::2013-03-18 07:59:56,473::task::957::TaskManager.Task::(_decref) Task=`18457ee7-f9ae-49fc-b7d1-040ca0e44d12`::ref 0 aborting False
>Thread-15733::DEBUG::2013-03-18 07:59:57,509::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15733::DEBUG::2013-03-18 07:59:57,510::task::568::TaskManager.Task::(_updateState) Task=`df7a20e8-cf69-4a11-b32d-7d960dca9fce`::moving from state init -> state preparing
>Thread-15733::INFO::2013-03-18 07:59:57,510::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15733::DEBUG::2013-03-18 07:59:57,510::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15733::DEBUG::2013-03-18 07:59:57,510::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15733::INFO::2013-03-18 07:59:57,511::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15733::DEBUG::2013-03-18 07:59:57,511::task::1151::TaskManager.Task::(prepare) Task=`df7a20e8-cf69-4a11-b32d-7d960dca9fce`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15733::DEBUG::2013-03-18 07:59:57,511::task::568::TaskManager.Task::(_updateState) Task=`df7a20e8-cf69-4a11-b32d-7d960dca9fce`::moving from state preparing -> state finished
>Thread-15733::DEBUG::2013-03-18 07:59:57,511::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15733::DEBUG::2013-03-18 07:59:57,511::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15733::DEBUG::2013-03-18 07:59:57,512::task::957::TaskManager.Task::(_decref) Task=`df7a20e8-cf69-4a11-b32d-7d960dca9fce`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 07:59:57,551::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:59:58,064::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 07:59:58,067::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 07:59:58,123::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00916593 s, 447 kB/s\n'; <rc> = 0
>Thread-15735::DEBUG::2013-03-18 07:59:58,537::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15735::DEBUG::2013-03-18 07:59:58,539::task::568::TaskManager.Task::(_updateState) Task=`eb0832f5-a1f2-4cdc-861e-c63c6547a571`::moving from state init -> state preparing
>Thread-15735::INFO::2013-03-18 07:59:58,540::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15735::DEBUG::2013-03-18 07:59:58,540::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15735::DEBUG::2013-03-18 07:59:58,541::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15735::INFO::2013-03-18 07:59:58,541::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15735::DEBUG::2013-03-18 07:59:58,541::task::1151::TaskManager.Task::(prepare) Task=`eb0832f5-a1f2-4cdc-861e-c63c6547a571`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15735::DEBUG::2013-03-18 07:59:58,542::task::568::TaskManager.Task::(_updateState) Task=`eb0832f5-a1f2-4cdc-861e-c63c6547a571`::moving from state preparing -> state finished
>Thread-15735::DEBUG::2013-03-18 07:59:58,542::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15735::DEBUG::2013-03-18 07:59:58,542::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15735::DEBUG::2013-03-18 07:59:58,544::task::957::TaskManager.Task::(_decref) Task=`eb0832f5-a1f2-4cdc-861e-c63c6547a571`::ref 0 aborting False
>Thread-15736::DEBUG::2013-03-18 07:59:59,564::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15736::DEBUG::2013-03-18 07:59:59,564::task::568::TaskManager.Task::(_updateState) Task=`31e45b56-4b4e-4a7e-b306-49a9573510a5`::moving from state init -> state preparing
>Thread-15736::INFO::2013-03-18 07:59:59,565::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15736::DEBUG::2013-03-18 07:59:59,565::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15736::DEBUG::2013-03-18 07:59:59,565::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15736::INFO::2013-03-18 07:59:59,565::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15736::DEBUG::2013-03-18 07:59:59,566::task::1151::TaskManager.Task::(prepare) Task=`31e45b56-4b4e-4a7e-b306-49a9573510a5`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15736::DEBUG::2013-03-18 07:59:59,566::task::568::TaskManager.Task::(_updateState) Task=`31e45b56-4b4e-4a7e-b306-49a9573510a5`::moving from state preparing -> state finished
>Thread-15736::DEBUG::2013-03-18 07:59:59,566::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15736::DEBUG::2013-03-18 07:59:59,566::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15736::DEBUG::2013-03-18 07:59:59,567::task::957::TaskManager.Task::(_decref) Task=`31e45b56-4b4e-4a7e-b306-49a9573510a5`::ref 0 aborting False
>Thread-15738::DEBUG::2013-03-18 08:00:00,589::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15738::DEBUG::2013-03-18 08:00:00,590::task::568::TaskManager.Task::(_updateState) Task=`a1da0845-2bc0-4115-a9e1-f701d02f6123`::moving from state init -> state preparing
>Thread-15738::INFO::2013-03-18 08:00:00,591::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15738::DEBUG::2013-03-18 08:00:00,591::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15738::DEBUG::2013-03-18 08:00:00,591::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15738::INFO::2013-03-18 08:00:00,592::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15738::DEBUG::2013-03-18 08:00:00,592::task::1151::TaskManager.Task::(prepare) Task=`a1da0845-2bc0-4115-a9e1-f701d02f6123`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15738::DEBUG::2013-03-18 08:00:00,592::task::568::TaskManager.Task::(_updateState) Task=`a1da0845-2bc0-4115-a9e1-f701d02f6123`::moving from state preparing -> state finished
>Thread-15738::DEBUG::2013-03-18 08:00:00,593::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15738::DEBUG::2013-03-18 08:00:00,593::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15738::DEBUG::2013-03-18 08:00:00,594::task::957::TaskManager.Task::(_decref) Task=`a1da0845-2bc0-4115-a9e1-f701d02f6123`::ref 0 aborting False
>Thread-15739::DEBUG::2013-03-18 08:00:01,633::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15739::DEBUG::2013-03-18 08:00:01,634::task::568::TaskManager.Task::(_updateState) Task=`03cb8609-bbab-4fe4-89f2-7f7db0cb30ce`::moving from state init -> state preparing
>Thread-15739::INFO::2013-03-18 08:00:01,634::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15739::DEBUG::2013-03-18 08:00:01,634::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15739::DEBUG::2013-03-18 08:00:01,635::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15739::INFO::2013-03-18 08:00:01,635::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15739::DEBUG::2013-03-18 08:00:01,635::task::1151::TaskManager.Task::(prepare) Task=`03cb8609-bbab-4fe4-89f2-7f7db0cb30ce`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15739::DEBUG::2013-03-18 08:00:01,635::task::568::TaskManager.Task::(_updateState) Task=`03cb8609-bbab-4fe4-89f2-7f7db0cb30ce`::moving from state preparing -> state finished
>Thread-15739::DEBUG::2013-03-18 08:00:01,636::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15739::DEBUG::2013-03-18 08:00:01,636::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15739::DEBUG::2013-03-18 08:00:01,637::task::957::TaskManager.Task::(_decref) Task=`03cb8609-bbab-4fe4-89f2-7f7db0cb30ce`::ref 0 aborting False
>Thread-15741::DEBUG::2013-03-18 08:00:02,671::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15741::DEBUG::2013-03-18 08:00:02,671::task::568::TaskManager.Task::(_updateState) Task=`c5d2d035-2754-4eb9-83cf-c3eb71c52871`::moving from state init -> state preparing
>Thread-15741::INFO::2013-03-18 08:00:02,672::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15741::DEBUG::2013-03-18 08:00:02,672::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15741::DEBUG::2013-03-18 08:00:02,672::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15741::INFO::2013-03-18 08:00:02,672::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15741::DEBUG::2013-03-18 08:00:02,673::task::1151::TaskManager.Task::(prepare) Task=`c5d2d035-2754-4eb9-83cf-c3eb71c52871`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15741::DEBUG::2013-03-18 08:00:02,673::task::568::TaskManager.Task::(_updateState) Task=`c5d2d035-2754-4eb9-83cf-c3eb71c52871`::moving from state preparing -> state finished
>Thread-15741::DEBUG::2013-03-18 08:00:02,673::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15741::DEBUG::2013-03-18 08:00:02,673::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15741::DEBUG::2013-03-18 08:00:02,673::task::957::TaskManager.Task::(_decref) Task=`c5d2d035-2754-4eb9-83cf-c3eb71c52871`::ref 0 aborting False
>Thread-15742::DEBUG::2013-03-18 08:00:03,709::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15742::DEBUG::2013-03-18 08:00:03,710::task::568::TaskManager.Task::(_updateState) Task=`76cfbf75-ae10-480a-b216-6de17518906c`::moving from state init -> state preparing
>Thread-15742::INFO::2013-03-18 08:00:03,710::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15742::DEBUG::2013-03-18 08:00:03,710::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15742::DEBUG::2013-03-18 08:00:03,710::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15742::INFO::2013-03-18 08:00:03,710::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15742::DEBUG::2013-03-18 08:00:03,711::task::1151::TaskManager.Task::(prepare) Task=`76cfbf75-ae10-480a-b216-6de17518906c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15742::DEBUG::2013-03-18 08:00:03,711::task::568::TaskManager.Task::(_updateState) Task=`76cfbf75-ae10-480a-b216-6de17518906c`::moving from state preparing -> state finished
>Thread-15742::DEBUG::2013-03-18 08:00:03,711::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15742::DEBUG::2013-03-18 08:00:03,711::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-22::DEBUG::2013-03-18 08:00:03,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-15742::DEBUG::2013-03-18 08:00:03,759::task::957::TaskManager.Task::(_decref) Task=`76cfbf75-ae10-480a-b216-6de17518906c`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 08:00:03,822::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091785 s, 4.5 MB/s\n'; <rc> = 0
>Thread-15744::DEBUG::2013-03-18 08:00:04,842::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15744::DEBUG::2013-03-18 08:00:04,843::task::568::TaskManager.Task::(_updateState) Task=`e5745d76-0584-4cc6-aa16-3edc022572d2`::moving from state init -> state preparing
>Thread-15744::INFO::2013-03-18 08:00:04,843::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15744::DEBUG::2013-03-18 08:00:04,844::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15744::DEBUG::2013-03-18 08:00:04,844::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15744::INFO::2013-03-18 08:00:04,844::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15744::DEBUG::2013-03-18 08:00:04,844::task::1151::TaskManager.Task::(prepare) Task=`e5745d76-0584-4cc6-aa16-3edc022572d2`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15744::DEBUG::2013-03-18 08:00:04,845::task::568::TaskManager.Task::(_updateState) Task=`e5745d76-0584-4cc6-aa16-3edc022572d2`::moving from state preparing -> state finished
>Thread-15744::DEBUG::2013-03-18 08:00:04,845::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15744::DEBUG::2013-03-18 08:00:04,845::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15744::DEBUG::2013-03-18 08:00:04,845::task::957::TaskManager.Task::(_decref) Task=`e5745d76-0584-4cc6-aa16-3edc022572d2`::ref 0 aborting False
>Thread-15745::DEBUG::2013-03-18 08:00:05,867::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15745::DEBUG::2013-03-18 08:00:05,868::task::568::TaskManager.Task::(_updateState) Task=`242f6b57-abff-4072-9909-4605d2a14186`::moving from state init -> state preparing
>Thread-15745::INFO::2013-03-18 08:00:05,868::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15745::DEBUG::2013-03-18 08:00:05,868::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15745::DEBUG::2013-03-18 08:00:05,869::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15745::INFO::2013-03-18 08:00:05,869::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15745::DEBUG::2013-03-18 08:00:05,869::task::1151::TaskManager.Task::(prepare) Task=`242f6b57-abff-4072-9909-4605d2a14186`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15745::DEBUG::2013-03-18 08:00:05,869::task::568::TaskManager.Task::(_updateState) Task=`242f6b57-abff-4072-9909-4605d2a14186`::moving from state preparing -> state finished
>Thread-15745::DEBUG::2013-03-18 08:00:05,870::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15745::DEBUG::2013-03-18 08:00:05,870::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15745::DEBUG::2013-03-18 08:00:05,870::task::957::TaskManager.Task::(_decref) Task=`242f6b57-abff-4072-9909-4605d2a14186`::ref 0 aborting False
>Thread-15746::DEBUG::2013-03-18 08:00:06,684::task::568::TaskManager.Task::(_updateState) Task=`28f6c86c-21c7-4420-9518-1e8fc3dc8933`::moving from state init -> state preparing
>Thread-15746::INFO::2013-03-18 08:00:06,684::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15746::INFO::2013-03-18 08:00:06,686::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0589611530304', 'lastCheck': '8.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '26.5', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.109912872314', 'lastCheck': '2.9', 'code': 0, 'valid': True}}
>Thread-15746::DEBUG::2013-03-18 08:00:06,686::task::1151::TaskManager.Task::(prepare) Task=`28f6c86c-21c7-4420-9518-1e8fc3dc8933`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0589611530304', 'lastCheck': '8.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '26.5', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.109912872314', 'lastCheck': '2.9', 'code': 0, 'valid': True}}
>Thread-15746::DEBUG::2013-03-18 08:00:06,687::task::568::TaskManager.Task::(_updateState) Task=`28f6c86c-21c7-4420-9518-1e8fc3dc8933`::moving from state preparing -> state finished
>Thread-15746::DEBUG::2013-03-18 08:00:06,687::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15746::DEBUG::2013-03-18 08:00:06,687::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15746::DEBUG::2013-03-18 08:00:06,688::task::957::TaskManager.Task::(_decref) Task=`28f6c86c-21c7-4420-9518-1e8fc3dc8933`::ref 0 aborting False
>Thread-15748::DEBUG::2013-03-18 08:00:06,890::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15748::DEBUG::2013-03-18 08:00:06,890::task::568::TaskManager.Task::(_updateState) Task=`6f23e2c2-99ff-4bb7-bdfa-980710436096`::moving from state init -> state preparing
>Thread-15748::INFO::2013-03-18 08:00:06,891::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15748::DEBUG::2013-03-18 08:00:06,891::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15748::DEBUG::2013-03-18 08:00:06,891::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15748::INFO::2013-03-18 08:00:06,892::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15748::DEBUG::2013-03-18 08:00:06,892::task::1151::TaskManager.Task::(prepare) Task=`6f23e2c2-99ff-4bb7-bdfa-980710436096`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15748::DEBUG::2013-03-18 08:00:06,892::task::568::TaskManager.Task::(_updateState) Task=`6f23e2c2-99ff-4bb7-bdfa-980710436096`::moving from state preparing -> state finished
>Thread-15748::DEBUG::2013-03-18 08:00:06,892::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15748::DEBUG::2013-03-18 08:00:06,893::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15748::DEBUG::2013-03-18 08:00:06,893::task::957::TaskManager.Task::(_decref) Task=`6f23e2c2-99ff-4bb7-bdfa-980710436096`::ref 0 aborting False
>Thread-15749::DEBUG::2013-03-18 08:00:07,917::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15749::DEBUG::2013-03-18 08:00:07,917::task::568::TaskManager.Task::(_updateState) Task=`15f2635e-2367-4675-be14-31aa2203d91e`::moving from state init -> state preparing
>Thread-15749::INFO::2013-03-18 08:00:07,917::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15749::DEBUG::2013-03-18 08:00:07,918::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15749::DEBUG::2013-03-18 08:00:07,918::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15749::INFO::2013-03-18 08:00:07,918::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15749::DEBUG::2013-03-18 08:00:07,918::task::1151::TaskManager.Task::(prepare) Task=`15f2635e-2367-4675-be14-31aa2203d91e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15749::DEBUG::2013-03-18 08:00:07,918::task::568::TaskManager.Task::(_updateState) Task=`15f2635e-2367-4675-be14-31aa2203d91e`::moving from state preparing -> state finished
>Thread-15749::DEBUG::2013-03-18 08:00:07,919::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15749::DEBUG::2013-03-18 08:00:07,919::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15749::DEBUG::2013-03-18 08:00:07,919::task::957::TaskManager.Task::(_decref) Task=`15f2635e-2367-4675-be14-31aa2203d91e`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 08:00:08,329::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:00:08,359::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000903262 s, 4.5 MB/s\n'; <rc> = 0
>Thread-15751::DEBUG::2013-03-18 08:00:08,947::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15751::DEBUG::2013-03-18 08:00:08,949::task::568::TaskManager.Task::(_updateState) Task=`ce75c1f5-5cc9-4c0d-b2f9-d4eddda2a733`::moving from state init -> state preparing
>Thread-15751::INFO::2013-03-18 08:00:08,950::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15751::DEBUG::2013-03-18 08:00:08,950::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15751::DEBUG::2013-03-18 08:00:08,951::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15751::INFO::2013-03-18 08:00:08,951::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15751::DEBUG::2013-03-18 08:00:08,951::task::1151::TaskManager.Task::(prepare) Task=`ce75c1f5-5cc9-4c0d-b2f9-d4eddda2a733`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15751::DEBUG::2013-03-18 08:00:08,952::task::568::TaskManager.Task::(_updateState) Task=`ce75c1f5-5cc9-4c0d-b2f9-d4eddda2a733`::moving from state preparing -> state finished
>Thread-15751::DEBUG::2013-03-18 08:00:08,952::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15751::DEBUG::2013-03-18 08:00:08,953::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15751::DEBUG::2013-03-18 08:00:08,953::task::957::TaskManager.Task::(_decref) Task=`ce75c1f5-5cc9-4c0d-b2f9-d4eddda2a733`::ref 0 aborting False
>Thread-15752::DEBUG::2013-03-18 08:00:10,036::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15752::DEBUG::2013-03-18 08:00:10,037::task::568::TaskManager.Task::(_updateState) Task=`fb14d0b7-0267-4713-be9e-e241a7204dfa`::moving from state init -> state preparing
>Thread-15752::INFO::2013-03-18 08:00:10,037::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15752::DEBUG::2013-03-18 08:00:10,037::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15752::DEBUG::2013-03-18 08:00:10,037::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15752::INFO::2013-03-18 08:00:10,038::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15752::DEBUG::2013-03-18 08:00:10,038::task::1151::TaskManager.Task::(prepare) Task=`fb14d0b7-0267-4713-be9e-e241a7204dfa`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15752::DEBUG::2013-03-18 08:00:10,038::task::568::TaskManager.Task::(_updateState) Task=`fb14d0b7-0267-4713-be9e-e241a7204dfa`::moving from state preparing -> state finished
>Thread-15752::DEBUG::2013-03-18 08:00:10,038::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15752::DEBUG::2013-03-18 08:00:10,038::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15752::DEBUG::2013-03-18 08:00:10,039::task::957::TaskManager.Task::(_decref) Task=`fb14d0b7-0267-4713-be9e-e241a7204dfa`::ref 0 aborting False
>Thread-15754::DEBUG::2013-03-18 08:00:11,067::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15754::DEBUG::2013-03-18 08:00:11,068::task::568::TaskManager.Task::(_updateState) Task=`0e1182e6-086a-4c93-8ca3-45d06b51037a`::moving from state init -> state preparing
>Thread-15754::INFO::2013-03-18 08:00:11,068::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15754::DEBUG::2013-03-18 08:00:11,068::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15754::DEBUG::2013-03-18 08:00:11,069::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15754::INFO::2013-03-18 08:00:11,069::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15754::DEBUG::2013-03-18 08:00:11,069::task::1151::TaskManager.Task::(prepare) Task=`0e1182e6-086a-4c93-8ca3-45d06b51037a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15754::DEBUG::2013-03-18 08:00:11,069::task::568::TaskManager.Task::(_updateState) Task=`0e1182e6-086a-4c93-8ca3-45d06b51037a`::moving from state preparing -> state finished
>Thread-15754::DEBUG::2013-03-18 08:00:11,070::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15754::DEBUG::2013-03-18 08:00:11,070::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15754::DEBUG::2013-03-18 08:00:11,071::task::957::TaskManager.Task::(_decref) Task=`0e1182e6-086a-4c93-8ca3-45d06b51037a`::ref 0 aborting False
>Thread-15755::DEBUG::2013-03-18 08:00:12,094::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15755::DEBUG::2013-03-18 08:00:12,095::task::568::TaskManager.Task::(_updateState) Task=`772bbb65-892e-44a2-8a02-7cccd46a5f9c`::moving from state init -> state preparing
>Thread-15755::INFO::2013-03-18 08:00:12,096::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15755::DEBUG::2013-03-18 08:00:12,096::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15755::DEBUG::2013-03-18 08:00:12,096::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15755::INFO::2013-03-18 08:00:12,096::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15755::DEBUG::2013-03-18 08:00:12,097::task::1151::TaskManager.Task::(prepare) Task=`772bbb65-892e-44a2-8a02-7cccd46a5f9c`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15755::DEBUG::2013-03-18 08:00:12,097::task::568::TaskManager.Task::(_updateState) Task=`772bbb65-892e-44a2-8a02-7cccd46a5f9c`::moving from state preparing -> state finished
>Thread-15755::DEBUG::2013-03-18 08:00:12,097::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15755::DEBUG::2013-03-18 08:00:12,097::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15755::DEBUG::2013-03-18 08:00:12,098::task::957::TaskManager.Task::(_decref) Task=`772bbb65-892e-44a2-8a02-7cccd46a5f9c`::ref 0 aborting False
>Thread-15757::DEBUG::2013-03-18 08:00:13,123::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15757::DEBUG::2013-03-18 08:00:13,124::task::568::TaskManager.Task::(_updateState) Task=`e723cb51-f0d8-496e-a80a-98f508d1c569`::moving from state init -> state preparing
>Thread-15757::INFO::2013-03-18 08:00:13,124::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15757::DEBUG::2013-03-18 08:00:13,124::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15757::DEBUG::2013-03-18 08:00:13,124::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15757::INFO::2013-03-18 08:00:13,125::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15757::DEBUG::2013-03-18 08:00:13,125::task::1151::TaskManager.Task::(prepare) Task=`e723cb51-f0d8-496e-a80a-98f508d1c569`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15757::DEBUG::2013-03-18 08:00:13,125::task::568::TaskManager.Task::(_updateState) Task=`e723cb51-f0d8-496e-a80a-98f508d1c569`::moving from state preparing -> state finished
>Thread-15757::DEBUG::2013-03-18 08:00:13,125::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15757::DEBUG::2013-03-18 08:00:13,125::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15757::DEBUG::2013-03-18 08:00:13,126::task::957::TaskManager.Task::(_decref) Task=`e723cb51-f0d8-496e-a80a-98f508d1c569`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 08:00:13,834::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:00:13,866::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000787315 s, 5.2 MB/s\n'; <rc> = 0
>Thread-15758::DEBUG::2013-03-18 08:00:14,159::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15758::DEBUG::2013-03-18 08:00:14,161::task::568::TaskManager.Task::(_updateState) Task=`2fb60f32-d0f4-4558-b4a8-06c4d3881474`::moving from state init -> state preparing
>Thread-15758::INFO::2013-03-18 08:00:14,162::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15758::DEBUG::2013-03-18 08:00:14,162::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15758::DEBUG::2013-03-18 08:00:14,162::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15758::INFO::2013-03-18 08:00:14,163::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15758::DEBUG::2013-03-18 08:00:14,163::task::1151::TaskManager.Task::(prepare) Task=`2fb60f32-d0f4-4558-b4a8-06c4d3881474`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15758::DEBUG::2013-03-18 08:00:14,163::task::568::TaskManager.Task::(_updateState) Task=`2fb60f32-d0f4-4558-b4a8-06c4d3881474`::moving from state preparing -> state finished
>Thread-15758::DEBUG::2013-03-18 08:00:14,164::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15758::DEBUG::2013-03-18 08:00:14,164::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15758::DEBUG::2013-03-18 08:00:14,164::task::957::TaskManager.Task::(_decref) Task=`2fb60f32-d0f4-4558-b4a8-06c4d3881474`::ref 0 aborting False
>Thread-15760::DEBUG::2013-03-18 08:00:15,229::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15760::DEBUG::2013-03-18 08:00:15,230::task::568::TaskManager.Task::(_updateState) Task=`328dbf34-4bad-4cb2-9acf-ed17b07009dd`::moving from state init -> state preparing
>Thread-15760::INFO::2013-03-18 08:00:15,230::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15760::DEBUG::2013-03-18 08:00:15,231::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15760::DEBUG::2013-03-18 08:00:15,231::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15760::INFO::2013-03-18 08:00:15,231::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15760::DEBUG::2013-03-18 08:00:15,232::task::1151::TaskManager.Task::(prepare) Task=`328dbf34-4bad-4cb2-9acf-ed17b07009dd`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15760::DEBUG::2013-03-18 08:00:15,232::task::568::TaskManager.Task::(_updateState) Task=`328dbf34-4bad-4cb2-9acf-ed17b07009dd`::moving from state preparing -> state finished
>Thread-15760::DEBUG::2013-03-18 08:00:15,232::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15760::DEBUG::2013-03-18 08:00:15,233::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15760::DEBUG::2013-03-18 08:00:15,233::task::957::TaskManager.Task::(_decref) Task=`328dbf34-4bad-4cb2-9acf-ed17b07009dd`::ref 0 aborting False
>Thread-15761::DEBUG::2013-03-18 08:00:16,262::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15761::DEBUG::2013-03-18 08:00:16,263::task::568::TaskManager.Task::(_updateState) Task=`2c7cbe9b-9fd3-4854-b583-6e0b8bfa74ef`::moving from state init -> state preparing
>Thread-15761::INFO::2013-03-18 08:00:16,263::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15761::DEBUG::2013-03-18 08:00:16,263::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15761::DEBUG::2013-03-18 08:00:16,263::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15761::INFO::2013-03-18 08:00:16,264::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15761::DEBUG::2013-03-18 08:00:16,264::task::1151::TaskManager.Task::(prepare) Task=`2c7cbe9b-9fd3-4854-b583-6e0b8bfa74ef`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15761::DEBUG::2013-03-18 08:00:16,264::task::568::TaskManager.Task::(_updateState) Task=`2c7cbe9b-9fd3-4854-b583-6e0b8bfa74ef`::moving from state preparing -> state finished
>Thread-15761::DEBUG::2013-03-18 08:00:16,265::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15761::DEBUG::2013-03-18 08:00:16,265::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15761::DEBUG::2013-03-18 08:00:16,265::task::957::TaskManager.Task::(_decref) Task=`2c7cbe9b-9fd3-4854-b583-6e0b8bfa74ef`::ref 0 aborting False
>Thread-15762::DEBUG::2013-03-18 08:00:16,905::task::568::TaskManager.Task::(_updateState) Task=`f94c7b49-8090-4f51-8c7b-b62cdf70693e`::moving from state init -> state preparing
>Thread-15762::INFO::2013-03-18 08:00:16,906::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15762::INFO::2013-03-18 08:00:16,906::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.032842874527', 'lastCheck': '8.5', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '36.7', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0343699455261', 'lastCheck': '3.0', 'code': 0, 'valid': True}}
>Thread-15762::DEBUG::2013-03-18 08:00:16,907::task::1151::TaskManager.Task::(prepare) Task=`f94c7b49-8090-4f51-8c7b-b62cdf70693e`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.032842874527', 'lastCheck': '8.5', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '36.7', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0343699455261', 'lastCheck': '3.0', 'code': 0, 'valid': True}}
>Thread-15762::DEBUG::2013-03-18 08:00:16,907::task::568::TaskManager.Task::(_updateState) Task=`f94c7b49-8090-4f51-8c7b-b62cdf70693e`::moving from state preparing -> state finished
>Thread-15762::DEBUG::2013-03-18 08:00:16,907::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15762::DEBUG::2013-03-18 08:00:16,907::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15762::DEBUG::2013-03-18 08:00:16,908::task::957::TaskManager.Task::(_decref) Task=`f94c7b49-8090-4f51-8c7b-b62cdf70693e`::ref 0 aborting False
>Thread-15764::DEBUG::2013-03-18 08:00:17,288::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15764::DEBUG::2013-03-18 08:00:17,288::task::568::TaskManager.Task::(_updateState) Task=`ac5fd85c-8ffe-4b8f-bb7f-046f6526d87b`::moving from state init -> state preparing
>Thread-15764::INFO::2013-03-18 08:00:17,290::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15764::DEBUG::2013-03-18 08:00:17,290::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15764::DEBUG::2013-03-18 08:00:17,291::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15764::INFO::2013-03-18 08:00:17,291::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15764::DEBUG::2013-03-18 08:00:17,291::task::1151::TaskManager.Task::(prepare) Task=`ac5fd85c-8ffe-4b8f-bb7f-046f6526d87b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15764::DEBUG::2013-03-18 08:00:17,291::task::568::TaskManager.Task::(_updateState) Task=`ac5fd85c-8ffe-4b8f-bb7f-046f6526d87b`::moving from state preparing -> state finished
>Thread-15764::DEBUG::2013-03-18 08:00:17,292::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15764::DEBUG::2013-03-18 08:00:17,292::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15764::DEBUG::2013-03-18 08:00:17,292::task::957::TaskManager.Task::(_decref) Task=`ac5fd85c-8ffe-4b8f-bb7f-046f6526d87b`::ref 0 aborting False
>Thread-15765::DEBUG::2013-03-18 08:00:18,312::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15765::DEBUG::2013-03-18 08:00:18,312::task::568::TaskManager.Task::(_updateState) Task=`f3e7c94c-f662-43cb-beef-2ce1a61dd15f`::moving from state init -> state preparing
>Thread-15765::INFO::2013-03-18 08:00:18,313::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15765::DEBUG::2013-03-18 08:00:18,313::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15765::DEBUG::2013-03-18 08:00:18,313::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15765::INFO::2013-03-18 08:00:18,313::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15765::DEBUG::2013-03-18 08:00:18,314::task::1151::TaskManager.Task::(prepare) Task=`f3e7c94c-f662-43cb-beef-2ce1a61dd15f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15765::DEBUG::2013-03-18 08:00:18,314::task::568::TaskManager.Task::(_updateState) Task=`f3e7c94c-f662-43cb-beef-2ce1a61dd15f`::moving from state preparing -> state finished
>Thread-15765::DEBUG::2013-03-18 08:00:18,314::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15765::DEBUG::2013-03-18 08:00:18,314::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15765::DEBUG::2013-03-18 08:00:18,314::task::957::TaskManager.Task::(_decref) Task=`f3e7c94c-f662-43cb-beef-2ce1a61dd15f`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 08:00:18,405::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:00:18,442::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000821736 s, 5.0 MB/s\n'; <rc> = 0
>Thread-15767::DEBUG::2013-03-18 08:00:19,340::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15767::DEBUG::2013-03-18 08:00:19,342::task::568::TaskManager.Task::(_updateState) Task=`174d5a31-8086-40b3-9dac-cef2207ca616`::moving from state init -> state preparing
>Thread-15767::INFO::2013-03-18 08:00:19,343::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15767::DEBUG::2013-03-18 08:00:19,344::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15767::DEBUG::2013-03-18 08:00:19,344::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15767::INFO::2013-03-18 08:00:19,344::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15767::DEBUG::2013-03-18 08:00:19,345::task::1151::TaskManager.Task::(prepare) Task=`174d5a31-8086-40b3-9dac-cef2207ca616`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15767::DEBUG::2013-03-18 08:00:19,345::task::568::TaskManager.Task::(_updateState) Task=`174d5a31-8086-40b3-9dac-cef2207ca616`::moving from state preparing -> state finished
>Thread-15767::DEBUG::2013-03-18 08:00:19,345::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15767::DEBUG::2013-03-18 08:00:19,345::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15767::DEBUG::2013-03-18 08:00:19,346::task::957::TaskManager.Task::(_decref) Task=`174d5a31-8086-40b3-9dac-cef2207ca616`::ref 0 aborting False
>Thread-15768::DEBUG::2013-03-18 08:00:20,370::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15768::DEBUG::2013-03-18 08:00:20,371::task::568::TaskManager.Task::(_updateState) Task=`661aa897-ade4-4707-bb69-a1778c35c320`::moving from state init -> state preparing
>Thread-15768::INFO::2013-03-18 08:00:20,371::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15768::DEBUG::2013-03-18 08:00:20,371::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15768::DEBUG::2013-03-18 08:00:20,372::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15768::INFO::2013-03-18 08:00:20,372::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15768::DEBUG::2013-03-18 08:00:20,372::task::1151::TaskManager.Task::(prepare) Task=`661aa897-ade4-4707-bb69-a1778c35c320`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15768::DEBUG::2013-03-18 08:00:20,372::task::568::TaskManager.Task::(_updateState) Task=`661aa897-ade4-4707-bb69-a1778c35c320`::moving from state preparing -> state finished
>Thread-15768::DEBUG::2013-03-18 08:00:20,373::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15768::DEBUG::2013-03-18 08:00:20,373::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15768::DEBUG::2013-03-18 08:00:20,373::task::957::TaskManager.Task::(_decref) Task=`661aa897-ade4-4707-bb69-a1778c35c320`::ref 0 aborting False
>Thread-15770::DEBUG::2013-03-18 08:00:21,393::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15770::DEBUG::2013-03-18 08:00:21,394::task::568::TaskManager.Task::(_updateState) Task=`5ac96fbc-aadc-4c69-ab35-86a97360604e`::moving from state init -> state preparing
>Thread-15770::INFO::2013-03-18 08:00:21,394::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15770::DEBUG::2013-03-18 08:00:21,394::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15770::DEBUG::2013-03-18 08:00:21,395::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15770::INFO::2013-03-18 08:00:21,395::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15770::DEBUG::2013-03-18 08:00:21,395::task::1151::TaskManager.Task::(prepare) Task=`5ac96fbc-aadc-4c69-ab35-86a97360604e`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15770::DEBUG::2013-03-18 08:00:21,395::task::568::TaskManager.Task::(_updateState) Task=`5ac96fbc-aadc-4c69-ab35-86a97360604e`::moving from state preparing -> state finished
>Thread-15770::DEBUG::2013-03-18 08:00:21,395::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15770::DEBUG::2013-03-18 08:00:21,396::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15770::DEBUG::2013-03-18 08:00:21,396::task::957::TaskManager.Task::(_decref) Task=`5ac96fbc-aadc-4c69-ab35-86a97360604e`::ref 0 aborting False
>Thread-15771::DEBUG::2013-03-18 08:00:22,419::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15771::DEBUG::2013-03-18 08:00:22,419::task::568::TaskManager.Task::(_updateState) Task=`c950b999-38e9-4e04-b800-ae085f536eb1`::moving from state init -> state preparing
>Thread-15771::INFO::2013-03-18 08:00:22,420::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15771::DEBUG::2013-03-18 08:00:22,420::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15771::DEBUG::2013-03-18 08:00:22,420::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15771::INFO::2013-03-18 08:00:22,421::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15771::DEBUG::2013-03-18 08:00:22,421::task::1151::TaskManager.Task::(prepare) Task=`c950b999-38e9-4e04-b800-ae085f536eb1`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15771::DEBUG::2013-03-18 08:00:22,421::task::568::TaskManager.Task::(_updateState) Task=`c950b999-38e9-4e04-b800-ae085f536eb1`::moving from state preparing -> state finished
>Thread-15771::DEBUG::2013-03-18 08:00:22,421::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15771::DEBUG::2013-03-18 08:00:22,422::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15771::DEBUG::2013-03-18 08:00:22,422::task::957::TaskManager.Task::(_decref) Task=`c950b999-38e9-4e04-b800-ae085f536eb1`::ref 0 aborting False
>Thread-15773::DEBUG::2013-03-18 08:00:23,446::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15773::DEBUG::2013-03-18 08:00:23,447::task::568::TaskManager.Task::(_updateState) Task=`1149f032-e3a3-407d-a0d2-d8d0ded0e6cf`::moving from state init -> state preparing
>Thread-15773::INFO::2013-03-18 08:00:23,447::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15773::DEBUG::2013-03-18 08:00:23,447::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15773::DEBUG::2013-03-18 08:00:23,448::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15773::INFO::2013-03-18 08:00:23,448::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15773::DEBUG::2013-03-18 08:00:23,448::task::1151::TaskManager.Task::(prepare) Task=`1149f032-e3a3-407d-a0d2-d8d0ded0e6cf`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15773::DEBUG::2013-03-18 08:00:23,448::task::568::TaskManager.Task::(_updateState) Task=`1149f032-e3a3-407d-a0d2-d8d0ded0e6cf`::moving from state preparing -> state finished
>Thread-15773::DEBUG::2013-03-18 08:00:23,449::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15773::DEBUG::2013-03-18 08:00:23,449::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15773::DEBUG::2013-03-18 08:00:23,449::task::957::TaskManager.Task::(_decref) Task=`1149f032-e3a3-407d-a0d2-d8d0ded0e6cf`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 08:00:23,871::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:00:23,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183841 s, 223 kB/s\n'; <rc> = 0
>Thread-15774::DEBUG::2013-03-18 08:00:24,484::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15774::DEBUG::2013-03-18 08:00:24,486::task::568::TaskManager.Task::(_updateState) Task=`d8d7a315-d20a-4ebc-932f-10c3966e9e81`::moving from state init -> state preparing
>Thread-15774::INFO::2013-03-18 08:00:24,487::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15774::DEBUG::2013-03-18 08:00:24,487::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15774::DEBUG::2013-03-18 08:00:24,488::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15774::INFO::2013-03-18 08:00:24,488::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15774::DEBUG::2013-03-18 08:00:24,488::task::1151::TaskManager.Task::(prepare) Task=`d8d7a315-d20a-4ebc-932f-10c3966e9e81`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15774::DEBUG::2013-03-18 08:00:24,488::task::568::TaskManager.Task::(_updateState) Task=`d8d7a315-d20a-4ebc-932f-10c3966e9e81`::moving from state preparing -> state finished
>Thread-15774::DEBUG::2013-03-18 08:00:24,489::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15774::DEBUG::2013-03-18 08:00:24,489::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15774::DEBUG::2013-03-18 08:00:24,489::task::957::TaskManager.Task::(_decref) Task=`d8d7a315-d20a-4ebc-932f-10c3966e9e81`::ref 0 aborting False
>Thread-15776::DEBUG::2013-03-18 08:00:25,514::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15776::DEBUG::2013-03-18 08:00:25,514::task::568::TaskManager.Task::(_updateState) Task=`cf212edf-7a5f-4fb4-a1b7-42b24371947f`::moving from state init -> state preparing
>Thread-15776::INFO::2013-03-18 08:00:25,515::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15776::DEBUG::2013-03-18 08:00:25,515::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15776::DEBUG::2013-03-18 08:00:25,515::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15776::INFO::2013-03-18 08:00:25,515::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15776::DEBUG::2013-03-18 08:00:25,515::task::1151::TaskManager.Task::(prepare) Task=`cf212edf-7a5f-4fb4-a1b7-42b24371947f`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15776::DEBUG::2013-03-18 08:00:25,516::task::568::TaskManager.Task::(_updateState) Task=`cf212edf-7a5f-4fb4-a1b7-42b24371947f`::moving from state preparing -> state finished
>Thread-15776::DEBUG::2013-03-18 08:00:25,516::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15776::DEBUG::2013-03-18 08:00:25,516::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15776::DEBUG::2013-03-18 08:00:25,516::task::957::TaskManager.Task::(_decref) Task=`cf212edf-7a5f-4fb4-a1b7-42b24371947f`::ref 0 aborting False
>Thread-15777::DEBUG::2013-03-18 08:00:26,542::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15777::DEBUG::2013-03-18 08:00:26,543::task::568::TaskManager.Task::(_updateState) Task=`14966185-3189-454b-9a6e-0a1ba43b0d3a`::moving from state init -> state preparing
>Thread-15777::INFO::2013-03-18 08:00:26,543::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15777::DEBUG::2013-03-18 08:00:26,543::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15777::DEBUG::2013-03-18 08:00:26,544::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15777::INFO::2013-03-18 08:00:26,544::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15777::DEBUG::2013-03-18 08:00:26,544::task::1151::TaskManager.Task::(prepare) Task=`14966185-3189-454b-9a6e-0a1ba43b0d3a`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15777::DEBUG::2013-03-18 08:00:26,545::task::568::TaskManager.Task::(_updateState) Task=`14966185-3189-454b-9a6e-0a1ba43b0d3a`::moving from state preparing -> state finished
>Thread-15777::DEBUG::2013-03-18 08:00:26,545::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15777::DEBUG::2013-03-18 08:00:26,545::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15777::DEBUG::2013-03-18 08:00:26,545::task::957::TaskManager.Task::(_decref) Task=`14966185-3189-454b-9a6e-0a1ba43b0d3a`::ref 0 aborting False
>Thread-15778::DEBUG::2013-03-18 08:00:27,135::task::568::TaskManager.Task::(_updateState) Task=`9d5b9b0d-8aa0-4f8b-81fc-0164a4f03375`::moving from state init -> state preparing
>Thread-15778::INFO::2013-03-18 08:00:27,136::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15778::INFO::2013-03-18 08:00:27,136::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0388910770416', 'lastCheck': '8.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '46.9', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0495309829712', 'lastCheck': '3.2', 'code': 0, 'valid': True}}
>Thread-15778::DEBUG::2013-03-18 08:00:27,137::task::1151::TaskManager.Task::(prepare) Task=`9d5b9b0d-8aa0-4f8b-81fc-0164a4f03375`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0388910770416', 'lastCheck': '8.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0', 'lastCheck': '46.9', 'code': 358, 'valid': False}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0495309829712', 'lastCheck': '3.2', 'code': 0, 'valid': True}}
>Thread-15778::DEBUG::2013-03-18 08:00:27,137::task::568::TaskManager.Task::(_updateState) Task=`9d5b9b0d-8aa0-4f8b-81fc-0164a4f03375`::moving from state preparing -> state finished
>Thread-15778::DEBUG::2013-03-18 08:00:27,137::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15778::DEBUG::2013-03-18 08:00:27,137::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15778::DEBUG::2013-03-18 08:00:27,137::task::957::TaskManager.Task::(_decref) Task=`9d5b9b0d-8aa0-4f8b-81fc-0164a4f03375`::ref 0 aborting False
>Thread-15715::DEBUG::2013-03-18 08:00:27,255::fileSD::131::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.1.147:_rhev_isos/848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-15715::DEBUG::2013-03-18 08:00:27,255::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-15715::DEBUG::2013-03-18 08:00:27,405::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Iso', 'DESCRIPTION=iso', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'REMOTE_PATH=10.34.1.147:/rhev/isos', 'ROLE=Regular', 'SDUUID=848fa27a-00f3-4bb4-b4da-101cd912b66e', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=82f0495b643c327dba06677f3f072dc015b7597c']
>Thread-15715::DEBUG::2013-03-18 08:00:27,411::fileSD::503::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-15715::WARNING::2013-03-18 08:00:27,412::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 848fa27a-00f3-4bb4-b4da-101cd912b66e_imageNS already registered
>Thread-15715::WARNING::2013-03-18 08:00:27,413::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 848fa27a-00f3-4bb4-b4da-101cd912b66e_volumeNS already registered
>Thread-15715::DEBUG::2013-03-18 08:00:27,414::sp::167::Storage.StoragePool::(_upgradePoolDomain) Preparing to upgrade domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 08:00:27,437::threadPool::33::Misc.ThreadPool::(__init__) Enter - numThreads: 5.0, waitTimeout: 3, maxTasks: 500.0
>Thread-15715::DEBUG::2013-03-18 08:00:27,435::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.upgrade_848fa27a-00f3-4bb4-b4da-101cd912b66e`ReqID=`6e19b3b8-56c6-4c6b-b901-ef4d900e3a88`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15715::DEBUG::2013-03-18 08:00:27,443::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.upgrade_848fa27a-00f3-4bb4-b4da-101cd912b66e' for lock type 'exclusive'
>Thread-15715::DEBUG::2013-03-18 08:00:27,446::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.upgrade_848fa27a-00f3-4bb4-b4da-101cd912b66e' is free. Now locking as 'exclusive' (1 active user)
>Thread-15715::DEBUG::2013-03-18 08:00:27,446::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.upgrade_848fa27a-00f3-4bb4-b4da-101cd912b66e`ReqID=`6e19b3b8-56c6-4c6b-b901-ef4d900e3a88`::Granted request
>Thread-15715::DEBUG::2013-03-18 08:00:27,451::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.848fa27a-00f3-4bb4-b4da-101cd912b66e`ReqID=`3cf457f0-b869-45f1-9b67-2bed95957425`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15715::DEBUG::2013-03-18 08:00:27,452::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.848fa27a-00f3-4bb4-b4da-101cd912b66e' for lock type 'exclusive'
>Thread-15715::DEBUG::2013-03-18 08:00:27,453::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.848fa27a-00f3-4bb4-b4da-101cd912b66e' is free. Now locking as 'exclusive' (1 active user)
>Thread-15715::DEBUG::2013-03-18 08:00:27,453::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.848fa27a-00f3-4bb4-b4da-101cd912b66e`ReqID=`3cf457f0-b869-45f1-9b67-2bed95957425`::Granted request
>Thread-15715::DEBUG::2013-03-18 08:00:27,458::sp::185::Storage.StoragePool::(_upgradePoolDomain) Domain `848fa27a-00f3-4bb4-b4da-101cd912b66e` is not a data domain it is an 2 domain, not upgrading
>Thread-15715::DEBUG::2013-03-18 08:00:27,458::sp::200::Storage.StoragePool::(_upgradePoolDomain) All domains are upgraded, unregistering from state change event
>Thread-15715::DEBUG::2013-03-18 08:00:27,461::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.848fa27a-00f3-4bb4-b4da-101cd912b66e'
>Thread-15715::DEBUG::2013-03-18 08:00:27,462::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.848fa27a-00f3-4bb4-b4da-101cd912b66e' (0 active users)
>Thread-15715::DEBUG::2013-03-18 08:00:27,463::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.848fa27a-00f3-4bb4-b4da-101cd912b66e' is free, finding out if anyone is waiting for it.
>Thread-15715::DEBUG::2013-03-18 08:00:27,467::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.848fa27a-00f3-4bb4-b4da-101cd912b66e', Clearing records.
>Thread-15715::DEBUG::2013-03-18 08:00:27,469::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.upgrade_848fa27a-00f3-4bb4-b4da-101cd912b66e'
>Thread-15715::DEBUG::2013-03-18 08:00:27,471::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.upgrade_848fa27a-00f3-4bb4-b4da-101cd912b66e' (0 active users)
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 08:00:27,475::storage_mailbox::504::Storage.MailBox.SpmMailMonitor::(__init__) SPM_MailMonitor - clearing outgoing mail, command is: ['dd', 'of=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/outbox', 'oflag=direct', 'iflag=fullblock', 'conv=notrunc', 'count=1']
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 08:00:27,477::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd of=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/outbox oflag=direct iflag=fullblock conv=notrunc count=1 bs=1024000' (cwd None)
>Thread-15715::DEBUG::2013-03-18 08:00:27,476::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.upgrade_848fa27a-00f3-4bb4-b4da-101cd912b66e' is free, finding out if anyone is waiting for it.
>Thread-15715::DEBUG::2013-03-18 08:00:27,479::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.upgrade_848fa27a-00f3-4bb4-b4da-101cd912b66e', Clearing records.
>Thread-21::DEBUG::2013-03-18 08:00:27,584::domainMonitor::216::Storage.DomainMonitorThread::(_monitorDomain) Domain 848fa27a-00f3-4bb4-b4da-101cd912b66e changed its status to Valid
>Thread-15785::DEBUG::2013-03-18 08:00:27,609::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15785::DEBUG::2013-03-18 08:00:27,613::task::568::TaskManager.Task::(_updateState) Task=`22693f53-89b3-4425-9931-6d04463d792b`::moving from state init -> state preparing
>Thread-15785::INFO::2013-03-18 08:00:27,615::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15785::DEBUG::2013-03-18 08:00:27,616::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15785::DEBUG::2013-03-18 08:00:27,616::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15785::INFO::2013-03-18 08:00:27,617::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15785::DEBUG::2013-03-18 08:00:27,617::task::1151::TaskManager.Task::(prepare) Task=`22693f53-89b3-4425-9931-6d04463d792b`::finished: {'taskStatus': {'code': 0, 'message': 'Task is initializing', 'taskState': 'running', 'taskResult': '', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15785::DEBUG::2013-03-18 08:00:27,618::task::568::TaskManager.Task::(_updateState) Task=`22693f53-89b3-4425-9931-6d04463d792b`::moving from state preparing -> state finished
>Thread-15785::DEBUG::2013-03-18 08:00:27,618::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15785::DEBUG::2013-03-18 08:00:27,619::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15785::DEBUG::2013-03-18 08:00:27,619::task::957::TaskManager.Task::(_decref) Task=`22693f53-89b3-4425-9931-6d04463d792b`::ref 0 aborting False
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 08:00:27,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.160269 s, 6.4 MB/s\n'; <rc> = 0
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 08:00:27,675::storage_mailbox::511::Storage.MailBox.SpmMailMonitor::(__init__) SPM_MailMonitor created for pool a1112cc0-b4af-4396-a3b6-2a3916293a6b
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 08:00:27,677::sp::302::Storage.StoragePool::(startSpm) SPM mailbox ready for pool a1112cc0-b4af-4396-a3b6-2a3916293a6b on master domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 08:00:27,678::sp::309::Storage.StoragePool::(startSpm) ended.
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 08:00:27,678::task::568::TaskManager.Task::(_updateState) Task=`b73b8c89-6250-482f-822a-45042a520f69`::moving from state running -> state finished
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 08:00:27,679::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 08:00:27,680::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 08:00:27,681::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 08:00:27,682::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 08:00:27,682::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Dummy-15786::DEBUG::2013-03-18 08:00:27,683::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 08:00:27,709::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 08:00:27,714::task::880::TaskManager.Task::(_runJobs) Task=`b73b8c89-6250-482f-822a-45042a520f69`::Task.run: exit - success: result 
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 08:00:27,715::task::957::TaskManager.Task::(_decref) Task=`b73b8c89-6250-482f-822a-45042a520f69`::ref 0 aborting False
>b73b8c89-6250-482f-822a-45042a520f69::DEBUG::2013-03-18 08:00:27,715::threadPool::55::Misc.ThreadPool::(setRunningTask) Number of running tasks: 0
>Dummy-15786::DEBUG::2013-03-18 08:00:27,868::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0902905 s, 11.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:00:28,489::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:00:28,520::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00299814 s, 1.4 MB/s\n'; <rc> = 0
>Thread-15787::DEBUG::2013-03-18 08:00:28,654::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15787::DEBUG::2013-03-18 08:00:28,658::task::568::TaskManager.Task::(_updateState) Task=`28f53f1a-aa54-4c64-bdea-bd1735646d3f`::moving from state init -> state preparing
>Thread-15787::INFO::2013-03-18 08:00:28,659::logUtils::37::dispatcher::(wrapper) Run and protect: getTaskStatus(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15787::DEBUG::2013-03-18 08:00:28,659::taskManager::93::TaskManager::(getTaskStatus) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15787::DEBUG::2013-03-18 08:00:28,660::taskManager::96::TaskManager::(getTaskStatus) Return. Response: {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}
>Thread-15787::INFO::2013-03-18 08:00:28,660::logUtils::39::dispatcher::(wrapper) Run and protect: getTaskStatus, Return response: {'taskStatus': {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15787::DEBUG::2013-03-18 08:00:28,660::task::1151::TaskManager.Task::(prepare) Task=`28f53f1a-aa54-4c64-bdea-bd1735646d3f`::finished: {'taskStatus': {'code': 0, 'message': '1 jobs completed successfully', 'taskState': 'finished', 'taskResult': 'success', 'taskID': 'b73b8c89-6250-482f-822a-45042a520f69'}}
>Thread-15787::DEBUG::2013-03-18 08:00:28,661::task::568::TaskManager.Task::(_updateState) Task=`28f53f1a-aa54-4c64-bdea-bd1735646d3f`::moving from state preparing -> state finished
>Thread-15787::DEBUG::2013-03-18 08:00:28,661::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15787::DEBUG::2013-03-18 08:00:28,661::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15787::DEBUG::2013-03-18 08:00:28,662::task::957::TaskManager.Task::(_decref) Task=`28f53f1a-aa54-4c64-bdea-bd1735646d3f`::ref 0 aborting False
>Thread-15788::DEBUG::2013-03-18 08:00:28,683::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15788::DEBUG::2013-03-18 08:00:28,684::task::568::TaskManager.Task::(_updateState) Task=`28f2019a-3039-4e7c-9df5-642dbe70e3e1`::moving from state init -> state preparing
>Thread-15788::INFO::2013-03-18 08:00:28,685::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15788::INFO::2013-03-18 08:00:28,686::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15788::DEBUG::2013-03-18 08:00:28,686::task::1151::TaskManager.Task::(prepare) Task=`28f2019a-3039-4e7c-9df5-642dbe70e3e1`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15788::DEBUG::2013-03-18 08:00:28,686::task::568::TaskManager.Task::(_updateState) Task=`28f2019a-3039-4e7c-9df5-642dbe70e3e1`::moving from state preparing -> state finished
>Thread-15788::DEBUG::2013-03-18 08:00:28,687::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15788::DEBUG::2013-03-18 08:00:28,687::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15788::DEBUG::2013-03-18 08:00:28,689::task::957::TaskManager.Task::(_decref) Task=`28f2019a-3039-4e7c-9df5-642dbe70e3e1`::ref 0 aborting False
>Thread-15789::DEBUG::2013-03-18 08:00:28,714::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15789::DEBUG::2013-03-18 08:00:28,715::task::568::TaskManager.Task::(_updateState) Task=`afdf2652-e13f-4adb-b3e2-ca1799428d37`::moving from state init -> state preparing
>Thread-15789::INFO::2013-03-18 08:00:28,715::logUtils::37::dispatcher::(wrapper) Run and protect: clearTask(taskID='b73b8c89-6250-482f-822a-45042a520f69', spUUID=None, options=None)
>Thread-15789::DEBUG::2013-03-18 08:00:28,716::taskManager::161::TaskManager::(clearTask) Entry. taskID: b73b8c89-6250-482f-822a-45042a520f69
>Thread-15789::DEBUG::2013-03-18 08:00:28,716::taskManager::166::TaskManager::(clearTask) Return.
>Thread-15789::INFO::2013-03-18 08:00:28,716::logUtils::39::dispatcher::(wrapper) Run and protect: clearTask, Return response: None
>Thread-15789::DEBUG::2013-03-18 08:00:28,717::task::1151::TaskManager.Task::(prepare) Task=`afdf2652-e13f-4adb-b3e2-ca1799428d37`::finished: None
>Thread-15789::DEBUG::2013-03-18 08:00:28,717::task::568::TaskManager.Task::(_updateState) Task=`afdf2652-e13f-4adb-b3e2-ca1799428d37`::moving from state preparing -> state finished
>Thread-15789::DEBUG::2013-03-18 08:00:28,717::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15789::DEBUG::2013-03-18 08:00:28,718::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15789::DEBUG::2013-03-18 08:00:28,718::task::957::TaskManager.Task::(_decref) Task=`afdf2652-e13f-4adb-b3e2-ca1799428d37`::ref 0 aborting False
>Thread-15790::DEBUG::2013-03-18 08:00:28,790::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15790::DEBUG::2013-03-18 08:00:28,791::task::568::TaskManager.Task::(_updateState) Task=`b70caaec-6d09-44e3-bc9b-ff3bbe40d70a`::moving from state init -> state preparing
>Thread-15790::INFO::2013-03-18 08:00:28,791::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15790::DEBUG::2013-03-18 08:00:28,793::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`5141057c-8a69-4a2c-9158-032e6cd64793`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15790::DEBUG::2013-03-18 08:00:28,793::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-15790::DEBUG::2013-03-18 08:00:28,794::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15790::DEBUG::2013-03-18 08:00:28,794::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`5141057c-8a69-4a2c-9158-032e6cd64793`::Granted request
>Thread-15790::DEBUG::2013-03-18 08:00:28,795::task::794::TaskManager.Task::(resourceAcquired) Task=`b70caaec-6d09-44e3-bc9b-ff3bbe40d70a`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-15790::DEBUG::2013-03-18 08:00:28,795::task::957::TaskManager.Task::(_decref) Task=`b70caaec-6d09-44e3-bc9b-ff3bbe40d70a`::ref 1 aborting False
>Thread-15790::INFO::2013-03-18 08:00:28,798::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15790::DEBUG::2013-03-18 08:00:28,799::task::1151::TaskManager.Task::(prepare) Task=`b70caaec-6d09-44e3-bc9b-ff3bbe40d70a`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15790::DEBUG::2013-03-18 08:00:28,799::task::568::TaskManager.Task::(_updateState) Task=`b70caaec-6d09-44e3-bc9b-ff3bbe40d70a`::moving from state preparing -> state finished
>Thread-15790::DEBUG::2013-03-18 08:00:28,800::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-15790::DEBUG::2013-03-18 08:00:28,800::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15790::DEBUG::2013-03-18 08:00:28,800::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15790::DEBUG::2013-03-18 08:00:28,801::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15790::DEBUG::2013-03-18 08:00:28,801::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15790::DEBUG::2013-03-18 08:00:28,801::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15790::DEBUG::2013-03-18 08:00:28,802::task::957::TaskManager.Task::(_decref) Task=`b70caaec-6d09-44e3-bc9b-ff3bbe40d70a`::ref 0 aborting False
>Thread-15791::DEBUG::2013-03-18 08:00:28,932::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15791::DEBUG::2013-03-18 08:00:28,933::task::568::TaskManager.Task::(_updateState) Task=`0329a4f5-4ea3-4a23-a5da-e990039e03ba`::moving from state init -> state preparing
>Thread-15791::INFO::2013-03-18 08:00:28,934::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksInfo(spUUID=None, options=None)
>Thread-15791::DEBUG::2013-03-18 08:00:28,934::taskManager::181::TaskManager::(getAllTasksInfo) Entry.
>Thread-15791::DEBUG::2013-03-18 08:00:28,934::taskManager::192::TaskManager::(getAllTasksInfo) Return. Response: {}
>Thread-15791::INFO::2013-03-18 08:00:28,935::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksInfo, Return response: {'allTasksInfo': {}}
>Thread-15791::DEBUG::2013-03-18 08:00:28,935::task::1151::TaskManager.Task::(prepare) Task=`0329a4f5-4ea3-4a23-a5da-e990039e03ba`::finished: {'allTasksInfo': {}}
>Thread-15791::DEBUG::2013-03-18 08:00:28,935::task::568::TaskManager.Task::(_updateState) Task=`0329a4f5-4ea3-4a23-a5da-e990039e03ba`::moving from state preparing -> state finished
>Thread-15791::DEBUG::2013-03-18 08:00:28,936::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15791::DEBUG::2013-03-18 08:00:28,936::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15791::DEBUG::2013-03-18 08:00:28,936::task::957::TaskManager.Task::(_decref) Task=`0329a4f5-4ea3-4a23-a5da-e990039e03ba`::ref 0 aborting False
>Thread-15792::DEBUG::2013-03-18 08:00:28,984::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15792::DEBUG::2013-03-18 08:00:28,985::task::568::TaskManager.Task::(_updateState) Task=`216d8e68-f285-49a5-8f49-54495229db61`::moving from state init -> state preparing
>Thread-15792::INFO::2013-03-18 08:00:28,985::logUtils::37::dispatcher::(wrapper) Run and protect: getAllTasksInfo(spUUID=None, options=None)
>Thread-15792::DEBUG::2013-03-18 08:00:28,986::taskManager::181::TaskManager::(getAllTasksInfo) Entry.
>Thread-15792::DEBUG::2013-03-18 08:00:28,986::taskManager::192::TaskManager::(getAllTasksInfo) Return. Response: {}
>Thread-15792::INFO::2013-03-18 08:00:28,986::logUtils::39::dispatcher::(wrapper) Run and protect: getAllTasksInfo, Return response: {'allTasksInfo': {}}
>Thread-15792::DEBUG::2013-03-18 08:00:28,987::task::1151::TaskManager.Task::(prepare) Task=`216d8e68-f285-49a5-8f49-54495229db61`::finished: {'allTasksInfo': {}}
>Thread-15792::DEBUG::2013-03-18 08:00:28,987::task::568::TaskManager.Task::(_updateState) Task=`216d8e68-f285-49a5-8f49-54495229db61`::moving from state preparing -> state finished
>Thread-15792::DEBUG::2013-03-18 08:00:28,987::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15792::DEBUG::2013-03-18 08:00:28,988::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15792::DEBUG::2013-03-18 08:00:28,988::task::957::TaskManager.Task::(_decref) Task=`216d8e68-f285-49a5-8f49-54495229db61`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:00:29,888::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:00:30,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921885 s, 11.1 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:00:32,098::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:00:32,252::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0541787 s, 18.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:00:33,925::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:00:33,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0108042 s, 379 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:00:34,276::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:00:34,445::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0682785 s, 15.0 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:00:36,465::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:00:36,639::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0684233 s, 15.0 MB/s\n'; <rc> = 0
>Thread-15797::DEBUG::2013-03-18 08:00:37,541::task::568::TaskManager.Task::(_updateState) Task=`f18eea7a-c375-487d-bb10-a59bf0496e27`::moving from state init -> state preparing
>Thread-15797::INFO::2013-03-18 08:00:37,542::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15797::INFO::2013-03-18 08:00:37,543::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0328719615936', 'lastCheck': '9.0', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00955414772034', 'lastCheck': '10.0', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0500609874725', 'lastCheck': '3.6', 'code': 0, 'valid': True}}
>Thread-15797::DEBUG::2013-03-18 08:00:37,543::task::1151::TaskManager.Task::(prepare) Task=`f18eea7a-c375-487d-bb10-a59bf0496e27`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0328719615936', 'lastCheck': '9.0', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00955414772034', 'lastCheck': '10.0', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0500609874725', 'lastCheck': '3.6', 'code': 0, 'valid': True}}
>Thread-15797::DEBUG::2013-03-18 08:00:37,543::task::568::TaskManager.Task::(_updateState) Task=`f18eea7a-c375-487d-bb10-a59bf0496e27`::moving from state preparing -> state finished
>Thread-15797::DEBUG::2013-03-18 08:00:37,544::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15797::DEBUG::2013-03-18 08:00:37,544::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15797::DEBUG::2013-03-18 08:00:37,544::task::957::TaskManager.Task::(_decref) Task=`f18eea7a-c375-487d-bb10-a59bf0496e27`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 08:00:38,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:00:38,598::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000878247 s, 4.7 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:00:38,660::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:00:38,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0879861 s, 11.6 MB/s\n'; <rc> = 0
>Thread-15799::DEBUG::2013-03-18 08:00:38,936::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15799::DEBUG::2013-03-18 08:00:38,938::task::568::TaskManager.Task::(_updateState) Task=`911ef8c8-98b4-4606-a604-b7d12d4c67c1`::moving from state init -> state preparing
>Thread-15799::INFO::2013-03-18 08:00:38,939::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15799::INFO::2013-03-18 08:00:38,940::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15799::DEBUG::2013-03-18 08:00:38,940::task::1151::TaskManager.Task::(prepare) Task=`911ef8c8-98b4-4606-a604-b7d12d4c67c1`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15799::DEBUG::2013-03-18 08:00:38,941::task::568::TaskManager.Task::(_updateState) Task=`911ef8c8-98b4-4606-a604-b7d12d4c67c1`::moving from state preparing -> state finished
>Thread-15799::DEBUG::2013-03-18 08:00:38,941::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15799::DEBUG::2013-03-18 08:00:38,941::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15799::DEBUG::2013-03-18 08:00:38,941::task::957::TaskManager.Task::(_decref) Task=`911ef8c8-98b4-4606-a604-b7d12d4c67c1`::ref 0 aborting False
>Thread-15800::DEBUG::2013-03-18 08:00:38,965::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15800::DEBUG::2013-03-18 08:00:38,966::task::568::TaskManager.Task::(_updateState) Task=`86504b0a-6754-4959-b131-cdbf539f4474`::moving from state init -> state preparing
>Thread-15800::INFO::2013-03-18 08:00:38,966::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15800::DEBUG::2013-03-18 08:00:38,967::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`d568ab61-bb11-4813-99d4-202288c2d970`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15800::DEBUG::2013-03-18 08:00:38,968::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-15800::DEBUG::2013-03-18 08:00:38,968::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15800::DEBUG::2013-03-18 08:00:38,969::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`d568ab61-bb11-4813-99d4-202288c2d970`::Granted request
>Thread-15800::DEBUG::2013-03-18 08:00:38,969::task::794::TaskManager.Task::(resourceAcquired) Task=`86504b0a-6754-4959-b131-cdbf539f4474`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-15800::DEBUG::2013-03-18 08:00:38,970::task::957::TaskManager.Task::(_decref) Task=`86504b0a-6754-4959-b131-cdbf539f4474`::ref 1 aborting False
>Thread-15800::INFO::2013-03-18 08:00:38,972::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15800::DEBUG::2013-03-18 08:00:38,972::task::1151::TaskManager.Task::(prepare) Task=`86504b0a-6754-4959-b131-cdbf539f4474`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15800::DEBUG::2013-03-18 08:00:38,973::task::568::TaskManager.Task::(_updateState) Task=`86504b0a-6754-4959-b131-cdbf539f4474`::moving from state preparing -> state finished
>Thread-15800::DEBUG::2013-03-18 08:00:38,973::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-15800::DEBUG::2013-03-18 08:00:38,973::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15800::DEBUG::2013-03-18 08:00:38,974::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15800::DEBUG::2013-03-18 08:00:38,974::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15800::DEBUG::2013-03-18 08:00:38,974::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15800::DEBUG::2013-03-18 08:00:38,975::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15800::DEBUG::2013-03-18 08:00:38,975::task::957::TaskManager.Task::(_decref) Task=`86504b0a-6754-4959-b131-cdbf539f4474`::ref 0 aborting False
>Thread-15802::DEBUG::2013-03-18 08:00:40,664::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15802::DEBUG::2013-03-18 08:00:40,665::task::568::TaskManager.Task::(_updateState) Task=`b32e53b2-f211-430e-9ddb-37917269eaf2`::moving from state init -> state preparing
>Thread-15802::INFO::2013-03-18 08:00:40,665::logUtils::37::dispatcher::(wrapper) Run and protect: getDeviceList(storageType=3, options={})
>Thread-15802::DEBUG::2013-03-18 08:00:40,666::misc::1025::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-15802::DEBUG::2013-03-18 08:00:40,666::misc::1027::SamplingMethod::(__call__) Got in to sampling method
>Thread-15802::DEBUG::2013-03-18 08:00:40,666::misc::1025::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-15802::DEBUG::2013-03-18 08:00:40,667::misc::1027::SamplingMethod::(__call__) Got in to sampling method
>Thread-15802::DEBUG::2013-03-18 08:00:40,667::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-15802::DEBUG::2013-03-18 08:00:40,728::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-15802::DEBUG::2013-03-18 08:00:40,731::misc::1035::SamplingMethod::(__call__) Returning last result
>MainProcess|Thread-15802::DEBUG::2013-03-18 08:00:40,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd of=/sys/class/scsi_host/host0/scan' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:00:40,866::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:00:41,051::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0855788 s, 12.0 MB/s\n'; <rc> = 0
>MainProcess|Thread-15802::DEBUG::2013-03-18 08:00:41,757::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd of=/sys/class/scsi_host/host1/scan' (cwd None)
>MainProcess|Thread-15802::DEBUG::2013-03-18 08:00:41,771::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd of=/sys/class/scsi_host/host2/scan' (cwd None)
>MainProcess|Thread-15802::DEBUG::2013-03-18 08:00:41,783::iscsi::388::Storage.ISCSI::(forceIScsiScan) Performing SCSI scan, this will take up to 30 seconds
>Dummy-15786::DEBUG::2013-03-18 08:00:43,072::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:00:43,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0806941 s, 12.7 MB/s\n'; <rc> = 0
>Thread-15802::DEBUG::2013-03-18 08:00:43,790::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
>Thread-15802::DEBUG::2013-03-18 08:00:43,903::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-15802::DEBUG::2013-03-18 08:00:43,906::lvm::482::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-15802::DEBUG::2013-03-18 08:00:43,906::lvm::484::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-15802::DEBUG::2013-03-18 08:00:43,907::lvm::493::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-15802::DEBUG::2013-03-18 08:00:43,907::lvm::495::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-15802::DEBUG::2013-03-18 08:00:43,908::lvm::513::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-15802::DEBUG::2013-03-18 08:00:43,908::lvm::515::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-15802::DEBUG::2013-03-18 08:00:43,909::misc::1035::SamplingMethod::(__call__) Returning last result
>Thread-15802::DEBUG::2013-03-18 08:00:43,909::lvm::339::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' got the operation mutex
>Thread-15802::DEBUG::2013-03-18 08:00:43,915::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm pvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,size,vg_name,vg_uuid,pe_start,pe_count,pe_alloc_count,mda_count,dev_size' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:00:43,981::lvm::373::OperationMutex::(_reloadvgs) Got the operational mutex
>Thread-22::DEBUG::2013-03-18 08:00:43,984::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-15802::DEBUG::2013-03-18 08:00:44,494::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:00:44,496::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-15802::DEBUG::2013-03-18 08:00:44,508::lvm::364::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 08:00:44,509::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>MainProcess|Thread-15802::DEBUG::2013-03-18 08:00:44,536::misc::83::Storage.Misc.excCmd::(<lambda>) '/sbin/dmsetup status' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:00:44,556::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0135724 s, 302 kB/s\n'; <rc> = 0
>MainProcess|Thread-15802::DEBUG::2013-03-18 08:00:44,562::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>MainProcess|Thread-15802::DEBUG::2013-03-18 08:00:44,570::misc::83::Storage.Misc.excCmd::(<lambda>) '/sbin/scsi_id --page=0x80 --whitelisted --export --replace-whitespace --device=/dev/dm-0' (cwd None)
>MainProcess|Thread-15802::DEBUG::2013-03-18 08:00:44,585::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>MainProcess|Thread-15802::DEBUG::2013-03-18 08:00:44,591::misc::83::Storage.Misc.excCmd::(<lambda>) '/sbin/scsi_id --page=0x80 --whitelisted --export --replace-whitespace --device=/dev/dm-10' (cwd None)
>MainProcess|Thread-15802::DEBUG::2013-03-18 08:00:44,603::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>MainProcess|Thread-15802::DEBUG::2013-03-18 08:00:44,614::misc::83::Storage.Misc.excCmd::(<lambda>) '/sbin/scsi_id --page=0x80 --whitelisted --export --replace-whitespace --device=/dev/dm-11' (cwd None)
>MainProcess|Thread-15802::DEBUG::2013-03-18 08:00:44,633::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>MainProcess|Thread-15802::DEBUG::2013-03-18 08:00:44,641::misc::83::Storage.Misc.excCmd::(<lambda>) '/sbin/scsi_id --page=0x80 --whitelisted --export --replace-whitespace --device=/dev/dm-25' (cwd None)
>MainProcess|Thread-15802::DEBUG::2013-03-18 08:00:44,654::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-15802::DEBUG::2013-03-18 08:00:44,659::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm pvcreate --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --test --metadatasize 128m --metadatacopies 2 --metadataignore y /dev/mapper/3600140558f280d707b243749b07609a3 /dev/mapper/36001405e2fe1e87706442878688ebb17 /dev/mapper/36001405398701597f844f4eb099342d0' (cwd None)
>Thread-15802::DEBUG::2013-03-18 08:00:45,188::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  TEST MODE: Metadata will NOT be updated and volumes will not be (de)activated.\n  Can\'t initialize physical volume "/dev/mapper/3600140558f280d707b243749b07609a3" of volume group "4bf968ef-020a-47ee-a1ae-f85eaeb8751a" without -ff\n  Can\'t initialize physical volume "/dev/mapper/36001405e2fe1e87706442878688ebb17" of volume group "f33e50cc-eb0a-4720-857d-bc7001a72567" without -ff\n'; <rc> = 5
>Thread-15802::DEBUG::2013-03-18 08:00:45,195::lvm::809::Storage.LVM::(testPVCreate) rc: 5, out: ['  Physical volume "/dev/mapper/36001405398701597f844f4eb099342d0" successfully created'], err: ['  TEST MODE: Metadata will NOT be updated and volumes will not be (de)activated.', '  Can\'t initialize physical volume "/dev/mapper/3600140558f280d707b243749b07609a3" of volume group "4bf968ef-020a-47ee-a1ae-f85eaeb8751a" without -ff', '  Can\'t initialize physical volume "/dev/mapper/36001405e2fe1e87706442878688ebb17" of volume group "f33e50cc-eb0a-4720-857d-bc7001a72567" without -ff'], unusedDevs: set(['/dev/mapper/36001405398701597f844f4eb099342d0']), usedDevs: set(['/dev/mapper/36001405e2fe1e87706442878688ebb17', '/dev/mapper/3600140558f280d707b243749b07609a3'])
>Thread-15802::INFO::2013-03-18 08:00:45,197::logUtils::39::dispatcher::(wrapper) Run and protect: getDeviceList, Return response: {'devList': [{'status': 'used', 'vendorID': 'LIO-ORG', 'capacity': '10737418240', 'fwrev': '4.0', 'vgUUID': 'AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'pathlist': [{'initiatorname': 'default', 'connection': '10.34.1.214', 'iqn': 'iqn.2003-01.org.linux-iscsi.dhcp-1-214.x8664:sn.113e745f4931', 'portal': '1', 'password': '******', 'port': '3260'}], 'logicalblocksize': '512', 'pathstatus': [{'physdev': 'sdc', 'type': 'iSCSI', 'state': 'active', 'lun': '1'}], 'devtype': 'iSCSI', 'physicalblocksize': '512', 'pvUUID': 'APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR', 'serial': 'SLIO-ORGIBLOCK', 'GUID': '3600140558f280d707b243749b07609a3', 'productID': 'IBLOCK'}, {'status': 'used', 'vendorID': 'LIO-ORG', 'capacity': '53687091200', 'fwrev': '4.0', 'vgUUID': '1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'pathlist': [{'initiatorname': 'default', 'connection': '10.34.1.214', 'iqn': 'iqn.2003-01.org.linux-iscsi.dhcp-1-214.x8664:sn.113e745f4931', 'portal': '1', 'password': '******', 'port': '3260'}], 'logicalblocksize': '512', 'pathstatus': [{'physdev': 'sdb', 'type': 'iSCSI', 'state': 'active', 'lun': '0'}], 'devtype': 'iSCSI', 'physicalblocksize': '512', 'pvUUID': '8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T', 'serial': 'SLIO-ORGIBLOCK', 'GUID': '36001405e2fe1e87706442878688ebb17', 'productID': 'IBLOCK'}, {'status': 'free', 'vendorID': 'LIO-ORG', 'capacity': '5368709120', 'fwrev': '4.0', 'vgUUID': '', 'pathlist': [{'initiatorname': 'default', 'connection': '10.34.1.214', 'iqn': 'iqn.2003-01.org.linux-iscsi.dhcp-1-214.x8664:sn.113e745f4931', 'portal': '1', 'password': '******', 'port': '3260'}], 'logicalblocksize': '512', 'pathstatus': [{'physdev': 'sdd', 'type': 'iSCSI', 'state': 'active', 'lun': '2'}], 'devtype': 'iSCSI', 'physicalblocksize': '512', 'pvUUID': '', 'serial': 'SLIO-ORGIBLOCK', 'GUID': '36001405398701597f844f4eb099342d0', 'productID': 'IBLOCK'}]}
>Thread-15802::DEBUG::2013-03-18 08:00:45,198::task::1151::TaskManager.Task::(prepare) Task=`b32e53b2-f211-430e-9ddb-37917269eaf2`::finished: {'devList': [{'status': 'used', 'fwrev': '4.0', 'vgUUID': 'AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'pathlist': [{'connection': '10.34.1.214', 'iqn': 'iqn.2003-01.org.linux-iscsi.dhcp-1-214.x8664:sn.113e745f4931', 'portal': '1', 'port': '3260', 'initiatorname': 'default'}], 'logicalblocksize': '512', 'devtype': 'iSCSI', 'physicalblocksize': '512', 'serial': 'SLIO-ORGIBLOCK', 'GUID': '3600140558f280d707b243749b07609a3', 'productID': 'IBLOCK', 'vendorID': 'LIO-ORG', 'capacity': '10737418240', 'pathstatus': [{'physdev': 'sdc', 'state': 'active', 'type': 'iSCSI', 'lun': '1'}], 'pvUUID': 'APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR'}, {'status': 'used', 'fwrev': '4.0', 'vgUUID': '1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'pathlist': [{'connection': '10.34.1.214', 'iqn': 'iqn.2003-01.org.linux-iscsi.dhcp-1-214.x8664:sn.113e745f4931', 'portal': '1', 'port': '3260', 'initiatorname': 'default'}], 'logicalblocksize': '512', 'devtype': 'iSCSI', 'physicalblocksize': '512', 'serial': 'SLIO-ORGIBLOCK', 'GUID': '36001405e2fe1e87706442878688ebb17', 'productID': 'IBLOCK', 'vendorID': 'LIO-ORG', 'capacity': '53687091200', 'pathstatus': [{'physdev': 'sdb', 'state': 'active', 'type': 'iSCSI', 'lun': '0'}], 'pvUUID': '8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T'}, {'status': 'free', 'fwrev': '4.0', 'vgUUID': '', 'pathlist': [{'connection': '10.34.1.214', 'iqn': 'iqn.2003-01.org.linux-iscsi.dhcp-1-214.x8664:sn.113e745f4931', 'portal': '1', 'port': '3260', 'initiatorname': 'default'}], 'logicalblocksize': '512', 'devtype': 'iSCSI', 'physicalblocksize': '512', 'serial': 'SLIO-ORGIBLOCK', 'GUID': '36001405398701597f844f4eb099342d0', 'productID': 'IBLOCK', 'vendorID': 'LIO-ORG', 'capacity': '5368709120', 'pathstatus': [{'physdev': 'sdd', 'state': 'active', 'type': 'iSCSI', 'lun': '2'}], 'pvUUID': ''}]}
>Thread-15802::DEBUG::2013-03-18 08:00:45,199::task::568::TaskManager.Task::(_updateState) Task=`b32e53b2-f211-430e-9ddb-37917269eaf2`::moving from state preparing -> state finished
>Thread-15802::DEBUG::2013-03-18 08:00:45,199::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15802::DEBUG::2013-03-18 08:00:45,199::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15802::DEBUG::2013-03-18 08:00:45,200::task::957::TaskManager.Task::(_decref) Task=`b32e53b2-f211-430e-9ddb-37917269eaf2`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:00:45,276::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:00:45,479::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0996882 s, 10.3 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:00:47,500::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Thread-15806::DEBUG::2013-03-18 08:00:47,798::task::568::TaskManager.Task::(_updateState) Task=`86fec22d-9230-4f1f-80ec-ba2afd8479f9`::moving from state init -> state preparing
>Thread-15806::INFO::2013-03-18 08:00:47,799::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15806::INFO::2013-03-18 08:00:47,800::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0351181030273', 'lastCheck': '9.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.01881980896', 'lastCheck': '0.0', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0530560016632', 'lastCheck': '3.2', 'code': 0, 'valid': True}}
>Thread-15806::DEBUG::2013-03-18 08:00:47,800::task::1151::TaskManager.Task::(prepare) Task=`86fec22d-9230-4f1f-80ec-ba2afd8479f9`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0351181030273', 'lastCheck': '9.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.01881980896', 'lastCheck': '0.0', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0530560016632', 'lastCheck': '3.2', 'code': 0, 'valid': True}}
>Thread-15806::DEBUG::2013-03-18 08:00:47,800::task::568::TaskManager.Task::(_updateState) Task=`86fec22d-9230-4f1f-80ec-ba2afd8479f9`::moving from state preparing -> state finished
>Thread-15806::DEBUG::2013-03-18 08:00:47,801::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15806::DEBUG::2013-03-18 08:00:47,801::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15806::DEBUG::2013-03-18 08:00:47,802::task::957::TaskManager.Task::(_decref) Task=`86fec22d-9230-4f1f-80ec-ba2afd8479f9`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:00:47,825::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0630291 s, 16.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:00:48,648::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 08:00:48,649::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-15808::DEBUG::2013-03-18 08:00:49,108::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15808::DEBUG::2013-03-18 08:00:49,111::task::568::TaskManager.Task::(_updateState) Task=`7324fc31-ba47-4840-870e-28b53192c8d2`::moving from state init -> state preparing
>Thread-15808::INFO::2013-03-18 08:00:49,112::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15808::INFO::2013-03-18 08:00:49,113::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15808::DEBUG::2013-03-18 08:00:49,114::task::1151::TaskManager.Task::(prepare) Task=`7324fc31-ba47-4840-870e-28b53192c8d2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15808::DEBUG::2013-03-18 08:00:49,114::task::568::TaskManager.Task::(_updateState) Task=`7324fc31-ba47-4840-870e-28b53192c8d2`::moving from state preparing -> state finished
>Thread-15808::DEBUG::2013-03-18 08:00:49,114::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15808::DEBUG::2013-03-18 08:00:49,114::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15808::DEBUG::2013-03-18 08:00:49,115::task::957::TaskManager.Task::(_decref) Task=`7324fc31-ba47-4840-870e-28b53192c8d2`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 08:00:49,131::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:00:49,137::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 08:00:49,139::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-15809::DEBUG::2013-03-18 08:00:49,167::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15809::DEBUG::2013-03-18 08:00:49,170::task::568::TaskManager.Task::(_updateState) Task=`080d57c0-1139-4435-b71b-aa8114d929a7`::moving from state init -> state preparing
>Thread-15809::INFO::2013-03-18 08:00:49,171::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15809::DEBUG::2013-03-18 08:00:49,185::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`960058af-7546-434f-a4cf-c597753ab511`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15809::DEBUG::2013-03-18 08:00:49,187::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-20::DEBUG::2013-03-18 08:00:49,193::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00758185 s, 540 kB/s\n'; <rc> = 0
>Thread-15809::DEBUG::2013-03-18 08:00:49,193::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15809::DEBUG::2013-03-18 08:00:49,203::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`960058af-7546-434f-a4cf-c597753ab511`::Granted request
>Thread-15809::DEBUG::2013-03-18 08:00:49,208::task::794::TaskManager.Task::(resourceAcquired) Task=`080d57c0-1139-4435-b71b-aa8114d929a7`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-15809::DEBUG::2013-03-18 08:00:49,211::task::957::TaskManager.Task::(_decref) Task=`080d57c0-1139-4435-b71b-aa8114d929a7`::ref 1 aborting False
>Thread-15809::INFO::2013-03-18 08:00:49,231::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15809::DEBUG::2013-03-18 08:00:49,234::task::1151::TaskManager.Task::(prepare) Task=`080d57c0-1139-4435-b71b-aa8114d929a7`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15809::DEBUG::2013-03-18 08:00:49,236::task::568::TaskManager.Task::(_updateState) Task=`080d57c0-1139-4435-b71b-aa8114d929a7`::moving from state preparing -> state finished
>Thread-15809::DEBUG::2013-03-18 08:00:49,238::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-15809::DEBUG::2013-03-18 08:00:49,240::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15809::DEBUG::2013-03-18 08:00:49,243::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15809::DEBUG::2013-03-18 08:00:49,245::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15809::DEBUG::2013-03-18 08:00:49,247::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15809::DEBUG::2013-03-18 08:00:49,249::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15809::DEBUG::2013-03-18 08:00:49,251::task::957::TaskManager.Task::(_decref) Task=`080d57c0-1139-4435-b71b-aa8114d929a7`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:00:49,846::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:00:50,089::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.14046 s, 7.3 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:00:52,109::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:00:52,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0821483 s, 12.5 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:00:54,310::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:00:54,509::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0970732 s, 10.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:00:54,565::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:00:54,621::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0166376 s, 246 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:00:56,530::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:00:56,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0803438 s, 12.7 MB/s\n'; <rc> = 0
>Thread-15814::DEBUG::2013-03-18 08:00:58,096::task::568::TaskManager.Task::(_updateState) Task=`d3b88ab7-191e-4cba-b63a-fa294b262b36`::moving from state init -> state preparing
>Thread-15814::INFO::2013-03-18 08:00:58,097::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15814::INFO::2013-03-18 08:00:58,098::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0570459365845', 'lastCheck': '8.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00821208953857', 'lastCheck': '0.3', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0583271980286', 'lastCheck': '3.5', 'code': 0, 'valid': True}}
>Thread-15814::DEBUG::2013-03-18 08:00:58,098::task::1151::TaskManager.Task::(prepare) Task=`d3b88ab7-191e-4cba-b63a-fa294b262b36`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0570459365845', 'lastCheck': '8.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00821208953857', 'lastCheck': '0.3', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0583271980286', 'lastCheck': '3.5', 'code': 0, 'valid': True}}
>Thread-15814::DEBUG::2013-03-18 08:00:58,098::task::568::TaskManager.Task::(_updateState) Task=`d3b88ab7-191e-4cba-b63a-fa294b262b36`::moving from state preparing -> state finished
>Thread-15814::DEBUG::2013-03-18 08:00:58,098::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15814::DEBUG::2013-03-18 08:00:58,099::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15814::DEBUG::2013-03-18 08:00:58,099::task::957::TaskManager.Task::(_decref) Task=`d3b88ab7-191e-4cba-b63a-fa294b262b36`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:00:58,729::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:00:58,904::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0792319 s, 12.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:00:59,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:00:59,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000780512 s, 5.2 MB/s\n'; <rc> = 0
>Thread-15816::DEBUG::2013-03-18 08:00:59,391::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15816::DEBUG::2013-03-18 08:00:59,393::task::568::TaskManager.Task::(_updateState) Task=`fa709bcc-2e8d-4f24-a361-3ff3195568b9`::moving from state init -> state preparing
>Thread-15816::INFO::2013-03-18 08:00:59,393::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15816::INFO::2013-03-18 08:00:59,394::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15816::DEBUG::2013-03-18 08:00:59,394::task::1151::TaskManager.Task::(prepare) Task=`fa709bcc-2e8d-4f24-a361-3ff3195568b9`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15816::DEBUG::2013-03-18 08:00:59,395::task::568::TaskManager.Task::(_updateState) Task=`fa709bcc-2e8d-4f24-a361-3ff3195568b9`::moving from state preparing -> state finished
>Thread-15816::DEBUG::2013-03-18 08:00:59,395::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15816::DEBUG::2013-03-18 08:00:59,395::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15816::DEBUG::2013-03-18 08:00:59,395::task::957::TaskManager.Task::(_decref) Task=`fa709bcc-2e8d-4f24-a361-3ff3195568b9`::ref 0 aborting False
>Thread-15817::DEBUG::2013-03-18 08:00:59,418::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15817::DEBUG::2013-03-18 08:00:59,419::task::568::TaskManager.Task::(_updateState) Task=`992963ac-7250-4769-9f25-6bce2345388b`::moving from state init -> state preparing
>Thread-15817::INFO::2013-03-18 08:00:59,419::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15817::DEBUG::2013-03-18 08:00:59,421::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`2a19e48a-cacc-4f7f-8f25-15a6d8bfc7e7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15817::DEBUG::2013-03-18 08:00:59,421::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-15817::DEBUG::2013-03-18 08:00:59,422::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15817::DEBUG::2013-03-18 08:00:59,422::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`2a19e48a-cacc-4f7f-8f25-15a6d8bfc7e7`::Granted request
>Thread-15817::DEBUG::2013-03-18 08:00:59,423::task::794::TaskManager.Task::(resourceAcquired) Task=`992963ac-7250-4769-9f25-6bce2345388b`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-15817::DEBUG::2013-03-18 08:00:59,424::task::957::TaskManager.Task::(_decref) Task=`992963ac-7250-4769-9f25-6bce2345388b`::ref 1 aborting False
>Thread-15817::INFO::2013-03-18 08:00:59,427::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15817::DEBUG::2013-03-18 08:00:59,427::task::1151::TaskManager.Task::(prepare) Task=`992963ac-7250-4769-9f25-6bce2345388b`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15817::DEBUG::2013-03-18 08:00:59,427::task::568::TaskManager.Task::(_updateState) Task=`992963ac-7250-4769-9f25-6bce2345388b`::moving from state preparing -> state finished
>Thread-15817::DEBUG::2013-03-18 08:00:59,428::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-15817::DEBUG::2013-03-18 08:00:59,428::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15817::DEBUG::2013-03-18 08:00:59,429::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15817::DEBUG::2013-03-18 08:00:59,429::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15817::DEBUG::2013-03-18 08:00:59,429::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15817::DEBUG::2013-03-18 08:00:59,430::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15817::DEBUG::2013-03-18 08:00:59,430::task::957::TaskManager.Task::(_decref) Task=`992963ac-7250-4769-9f25-6bce2345388b`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:01:00,925::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:01:01,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0774797 s, 13.2 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:01:03,136::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:01:03,335::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.10021 s, 10.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:01:04,627::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:01:04,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118114 s, 347 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:01:05,356::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:01:05,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0847803 s, 12.1 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:01:07,557::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:01:07,743::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0719533 s, 14.2 MB/s\n'; <rc> = 0
>Thread-15822::DEBUG::2013-03-18 08:01:08,360::task::568::TaskManager.Task::(_updateState) Task=`f05b12ed-ef13-4295-ab74-b8dc9f6a634a`::moving from state init -> state preparing
>Thread-15822::INFO::2013-03-18 08:01:08,361::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15822::INFO::2013-03-18 08:01:08,361::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0283839702606', 'lastCheck': '9.0', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00922012329102', 'lastCheck': '0.5', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0434100627899', 'lastCheck': '3.7', 'code': 0, 'valid': True}}
>Thread-15822::DEBUG::2013-03-18 08:01:08,362::task::1151::TaskManager.Task::(prepare) Task=`f05b12ed-ef13-4295-ab74-b8dc9f6a634a`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0283839702606', 'lastCheck': '9.0', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00922012329102', 'lastCheck': '0.5', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0434100627899', 'lastCheck': '3.7', 'code': 0, 'valid': True}}
>Thread-15822::DEBUG::2013-03-18 08:01:08,362::task::568::TaskManager.Task::(_updateState) Task=`f05b12ed-ef13-4295-ab74-b8dc9f6a634a`::moving from state preparing -> state finished
>Thread-15822::DEBUG::2013-03-18 08:01:08,363::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15822::DEBUG::2013-03-18 08:01:08,363::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15822::DEBUG::2013-03-18 08:01:08,363::task::957::TaskManager.Task::(_decref) Task=`f05b12ed-ef13-4295-ab74-b8dc9f6a634a`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 08:01:09,388::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:01:09,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00845168 s, 485 kB/s\n'; <rc> = 0
>Thread-15824::DEBUG::2013-03-18 08:01:09,553::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15824::DEBUG::2013-03-18 08:01:09,555::task::568::TaskManager.Task::(_updateState) Task=`57966de3-d2d9-4bbb-bd8d-7aeec50bd7b4`::moving from state init -> state preparing
>Thread-15824::INFO::2013-03-18 08:01:09,556::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15824::INFO::2013-03-18 08:01:09,557::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15824::DEBUG::2013-03-18 08:01:09,557::task::1151::TaskManager.Task::(prepare) Task=`57966de3-d2d9-4bbb-bd8d-7aeec50bd7b4`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15824::DEBUG::2013-03-18 08:01:09,558::task::568::TaskManager.Task::(_updateState) Task=`57966de3-d2d9-4bbb-bd8d-7aeec50bd7b4`::moving from state preparing -> state finished
>Thread-15824::DEBUG::2013-03-18 08:01:09,558::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15824::DEBUG::2013-03-18 08:01:09,558::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15824::DEBUG::2013-03-18 08:01:09,559::task::957::TaskManager.Task::(_decref) Task=`57966de3-d2d9-4bbb-bd8d-7aeec50bd7b4`::ref 0 aborting False
>Thread-15825::DEBUG::2013-03-18 08:01:09,582::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15825::DEBUG::2013-03-18 08:01:09,583::task::568::TaskManager.Task::(_updateState) Task=`f9ef6b08-b89a-491e-ad35-fd2340838a16`::moving from state init -> state preparing
>Thread-15825::INFO::2013-03-18 08:01:09,583::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15825::DEBUG::2013-03-18 08:01:09,585::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`975c1521-15aa-4e20-99e7-664fbaba9942`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15825::DEBUG::2013-03-18 08:01:09,585::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-15825::DEBUG::2013-03-18 08:01:09,586::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15825::DEBUG::2013-03-18 08:01:09,586::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`975c1521-15aa-4e20-99e7-664fbaba9942`::Granted request
>Thread-15825::DEBUG::2013-03-18 08:01:09,587::task::794::TaskManager.Task::(resourceAcquired) Task=`f9ef6b08-b89a-491e-ad35-fd2340838a16`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-15825::DEBUG::2013-03-18 08:01:09,588::task::957::TaskManager.Task::(_decref) Task=`f9ef6b08-b89a-491e-ad35-fd2340838a16`::ref 1 aborting False
>Thread-15825::INFO::2013-03-18 08:01:09,591::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15825::DEBUG::2013-03-18 08:01:09,591::task::1151::TaskManager.Task::(prepare) Task=`f9ef6b08-b89a-491e-ad35-fd2340838a16`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15825::DEBUG::2013-03-18 08:01:09,591::task::568::TaskManager.Task::(_updateState) Task=`f9ef6b08-b89a-491e-ad35-fd2340838a16`::moving from state preparing -> state finished
>Thread-15825::DEBUG::2013-03-18 08:01:09,592::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-15825::DEBUG::2013-03-18 08:01:09,592::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15825::DEBUG::2013-03-18 08:01:09,593::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15825::DEBUG::2013-03-18 08:01:09,593::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15825::DEBUG::2013-03-18 08:01:09,593::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15825::DEBUG::2013-03-18 08:01:09,593::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15825::DEBUG::2013-03-18 08:01:09,594::task::957::TaskManager.Task::(_decref) Task=`f9ef6b08-b89a-491e-ad35-fd2340838a16`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:01:09,765::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:01:09,948::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0854182 s, 12.0 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:01:11,969::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:01:12,154::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.085872 s, 11.9 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:01:14,175::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:01:14,372::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0983191 s, 10.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:01:14,683::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:01:14,725::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0126641 s, 323 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:01:16,393::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:01:16,554::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0581351 s, 17.6 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:01:18,579::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Thread-15830::DEBUG::2013-03-18 08:01:18,697::task::568::TaskManager.Task::(_updateState) Task=`f13d65bb-7749-4179-b493-097fb1e90aa1`::moving from state init -> state preparing
>Thread-15830::INFO::2013-03-18 08:01:18,705::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15830::INFO::2013-03-18 08:01:18,711::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0511901378632', 'lastCheck': '9.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00919699668884', 'lastCheck': '0.8', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0441591739655', 'lastCheck': '4.0', 'code': 0, 'valid': True}}
>Thread-15830::DEBUG::2013-03-18 08:01:18,712::task::1151::TaskManager.Task::(prepare) Task=`f13d65bb-7749-4179-b493-097fb1e90aa1`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0511901378632', 'lastCheck': '9.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00919699668884', 'lastCheck': '0.8', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0441591739655', 'lastCheck': '4.0', 'code': 0, 'valid': True}}
>Thread-15830::DEBUG::2013-03-18 08:01:18,713::task::568::TaskManager.Task::(_updateState) Task=`f13d65bb-7749-4179-b493-097fb1e90aa1`::moving from state preparing -> state finished
>Thread-15830::DEBUG::2013-03-18 08:01:18,715::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15830::DEBUG::2013-03-18 08:01:18,716::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15830::DEBUG::2013-03-18 08:01:18,718::task::957::TaskManager.Task::(_decref) Task=`f13d65bb-7749-4179-b493-097fb1e90aa1`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:01:18,853::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.158192 s, 6.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:01:19,485::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:01:19,513::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000844132 s, 4.9 MB/s\n'; <rc> = 0
>Thread-15832::DEBUG::2013-03-18 08:01:19,712::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15832::DEBUG::2013-03-18 08:01:19,714::task::568::TaskManager.Task::(_updateState) Task=`cbc756b9-f324-4890-b98e-339341f0ffca`::moving from state init -> state preparing
>Thread-15832::INFO::2013-03-18 08:01:19,715::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15832::INFO::2013-03-18 08:01:19,716::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15832::DEBUG::2013-03-18 08:01:19,716::task::1151::TaskManager.Task::(prepare) Task=`cbc756b9-f324-4890-b98e-339341f0ffca`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15832::DEBUG::2013-03-18 08:01:19,717::task::568::TaskManager.Task::(_updateState) Task=`cbc756b9-f324-4890-b98e-339341f0ffca`::moving from state preparing -> state finished
>Thread-15832::DEBUG::2013-03-18 08:01:19,717::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15832::DEBUG::2013-03-18 08:01:19,717::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15832::DEBUG::2013-03-18 08:01:19,718::task::957::TaskManager.Task::(_decref) Task=`cbc756b9-f324-4890-b98e-339341f0ffca`::ref 0 aborting False
>Thread-15833::DEBUG::2013-03-18 08:01:19,748::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15833::DEBUG::2013-03-18 08:01:19,749::task::568::TaskManager.Task::(_updateState) Task=`ad7a445c-1719-4e77-a00a-dbcb61661910`::moving from state init -> state preparing
>Thread-15833::INFO::2013-03-18 08:01:19,750::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15833::DEBUG::2013-03-18 08:01:19,751::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`8285cc34-96d8-4238-8abc-ba521c9fdc38`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15833::DEBUG::2013-03-18 08:01:19,751::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-15833::DEBUG::2013-03-18 08:01:19,752::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15833::DEBUG::2013-03-18 08:01:19,753::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`8285cc34-96d8-4238-8abc-ba521c9fdc38`::Granted request
>Thread-15833::DEBUG::2013-03-18 08:01:19,753::task::794::TaskManager.Task::(resourceAcquired) Task=`ad7a445c-1719-4e77-a00a-dbcb61661910`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-15833::DEBUG::2013-03-18 08:01:19,754::task::957::TaskManager.Task::(_decref) Task=`ad7a445c-1719-4e77-a00a-dbcb61661910`::ref 1 aborting False
>Thread-15833::INFO::2013-03-18 08:01:19,756::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15833::DEBUG::2013-03-18 08:01:19,757::task::1151::TaskManager.Task::(prepare) Task=`ad7a445c-1719-4e77-a00a-dbcb61661910`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15833::DEBUG::2013-03-18 08:01:19,757::task::568::TaskManager.Task::(_updateState) Task=`ad7a445c-1719-4e77-a00a-dbcb61661910`::moving from state preparing -> state finished
>Thread-15833::DEBUG::2013-03-18 08:01:19,758::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-15833::DEBUG::2013-03-18 08:01:19,758::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15833::DEBUG::2013-03-18 08:01:19,758::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15833::DEBUG::2013-03-18 08:01:19,759::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15833::DEBUG::2013-03-18 08:01:19,759::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15833::DEBUG::2013-03-18 08:01:19,759::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15833::DEBUG::2013-03-18 08:01:19,760::task::957::TaskManager.Task::(_decref) Task=`ad7a445c-1719-4e77-a00a-dbcb61661910`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:01:20,873::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:01:21,044::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.071278 s, 14.4 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:01:23,066::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:01:23,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0818594 s, 12.5 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:01:24,731::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:01:24,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000894072 s, 4.6 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:01:25,269::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:01:25,447::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0826422 s, 12.4 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:01:27,468::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:01:27,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0980494 s, 10.4 MB/s\n'; <rc> = 0
>Thread-15838::DEBUG::2013-03-18 08:01:28,996::task::568::TaskManager.Task::(_updateState) Task=`3bb37bd0-e7f5-4837-a8d7-08face0c5be0`::moving from state init -> state preparing
>Thread-15838::INFO::2013-03-18 08:01:28,997::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15838::INFO::2013-03-18 08:01:28,997::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0296671390533', 'lastCheck': '9.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0104489326477', 'lastCheck': '1.0', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0351071357727', 'lastCheck': '4.2', 'code': 0, 'valid': True}}
>Thread-15838::DEBUG::2013-03-18 08:01:28,997::task::1151::TaskManager.Task::(prepare) Task=`3bb37bd0-e7f5-4837-a8d7-08face0c5be0`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0296671390533', 'lastCheck': '9.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0104489326477', 'lastCheck': '1.0', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0351071357727', 'lastCheck': '4.2', 'code': 0, 'valid': True}}
>Thread-15838::DEBUG::2013-03-18 08:01:28,998::task::568::TaskManager.Task::(_updateState) Task=`3bb37bd0-e7f5-4837-a8d7-08face0c5be0`::moving from state preparing -> state finished
>Thread-15838::DEBUG::2013-03-18 08:01:28,999::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15838::DEBUG::2013-03-18 08:01:28,999::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15838::DEBUG::2013-03-18 08:01:28,999::task::957::TaskManager.Task::(_decref) Task=`3bb37bd0-e7f5-4837-a8d7-08face0c5be0`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 08:01:29,559::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:01:29,593::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00617028 s, 664 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:01:29,685::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Thread-15840::DEBUG::2013-03-18 08:01:29,913::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Dummy-15786::DEBUG::2013-03-18 08:01:29,917::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0984139 s, 10.4 MB/s\n'; <rc> = 0
>Thread-15840::DEBUG::2013-03-18 08:01:29,936::task::568::TaskManager.Task::(_updateState) Task=`b8e6d71e-55c7-4feb-9259-e60b521d0765`::moving from state init -> state preparing
>Thread-15840::INFO::2013-03-18 08:01:29,943::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15840::INFO::2013-03-18 08:01:29,948::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15840::DEBUG::2013-03-18 08:01:29,949::task::1151::TaskManager.Task::(prepare) Task=`b8e6d71e-55c7-4feb-9259-e60b521d0765`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15840::DEBUG::2013-03-18 08:01:29,950::task::568::TaskManager.Task::(_updateState) Task=`b8e6d71e-55c7-4feb-9259-e60b521d0765`::moving from state preparing -> state finished
>Thread-15840::DEBUG::2013-03-18 08:01:29,952::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15840::DEBUG::2013-03-18 08:01:29,953::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15840::DEBUG::2013-03-18 08:01:29,954::task::957::TaskManager.Task::(_decref) Task=`b8e6d71e-55c7-4feb-9259-e60b521d0765`::ref 0 aborting False
>Thread-15841::DEBUG::2013-03-18 08:01:29,976::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15841::DEBUG::2013-03-18 08:01:29,978::task::568::TaskManager.Task::(_updateState) Task=`110b7b46-2872-4a00-8eee-89777c1e9dbc`::moving from state init -> state preparing
>Thread-15841::INFO::2013-03-18 08:01:29,980::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15841::DEBUG::2013-03-18 08:01:29,987::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`0b878bd9-258c-4816-8050-f2051f2fbab7`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15841::DEBUG::2013-03-18 08:01:29,988::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-15841::DEBUG::2013-03-18 08:01:29,991::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15841::DEBUG::2013-03-18 08:01:29,992::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`0b878bd9-258c-4816-8050-f2051f2fbab7`::Granted request
>Thread-15841::DEBUG::2013-03-18 08:01:29,995::task::794::TaskManager.Task::(resourceAcquired) Task=`110b7b46-2872-4a00-8eee-89777c1e9dbc`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-15841::DEBUG::2013-03-18 08:01:29,996::task::957::TaskManager.Task::(_decref) Task=`110b7b46-2872-4a00-8eee-89777c1e9dbc`::ref 1 aborting False
>Thread-15841::INFO::2013-03-18 08:01:30,007::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2338324480', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15841::DEBUG::2013-03-18 08:01:30,008::task::1151::TaskManager.Task::(prepare) Task=`110b7b46-2872-4a00-8eee-89777c1e9dbc`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2338324480', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15841::DEBUG::2013-03-18 08:01:30,009::task::568::TaskManager.Task::(_updateState) Task=`110b7b46-2872-4a00-8eee-89777c1e9dbc`::moving from state preparing -> state finished
>Thread-15841::DEBUG::2013-03-18 08:01:30,010::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-15841::DEBUG::2013-03-18 08:01:30,011::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15841::DEBUG::2013-03-18 08:01:30,012::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15841::DEBUG::2013-03-18 08:01:30,014::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15841::DEBUG::2013-03-18 08:01:30,015::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15841::DEBUG::2013-03-18 08:01:30,016::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15841::DEBUG::2013-03-18 08:01:30,017::task::957::TaskManager.Task::(_decref) Task=`110b7b46-2872-4a00-8eee-89777c1e9dbc`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:01:32,022::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:01:32,202::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0802847 s, 12.8 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:01:34,222::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:01:34,405::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0835703 s, 12.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:01:34,772::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:01:34,831::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0199767 s, 205 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:01:36,427::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:01:36,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.106257 s, 9.6 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:01:38,662::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:01:38,840::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0746422 s, 13.7 MB/s\n'; <rc> = 0
>Thread-15846::DEBUG::2013-03-18 08:01:39,268::task::568::TaskManager.Task::(_updateState) Task=`dc489f29-2c99-4ee1-833c-0b49f2648ea9`::moving from state init -> state preparing
>Thread-15846::INFO::2013-03-18 08:01:39,269::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15846::INFO::2013-03-18 08:01:39,269::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0373771190643', 'lastCheck': '9.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0102298259735', 'lastCheck': '1.2', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.061418056488', 'lastCheck': '4.4', 'code': 0, 'valid': True}}
>Thread-15846::DEBUG::2013-03-18 08:01:39,270::task::1151::TaskManager.Task::(prepare) Task=`dc489f29-2c99-4ee1-833c-0b49f2648ea9`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0373771190643', 'lastCheck': '9.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0102298259735', 'lastCheck': '1.2', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.061418056488', 'lastCheck': '4.4', 'code': 0, 'valid': True}}
>Thread-15846::DEBUG::2013-03-18 08:01:39,270::task::568::TaskManager.Task::(_updateState) Task=`dc489f29-2c99-4ee1-833c-0b49f2648ea9`::moving from state preparing -> state finished
>Thread-15846::DEBUG::2013-03-18 08:01:39,270::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15846::DEBUG::2013-03-18 08:01:39,271::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15846::DEBUG::2013-03-18 08:01:39,271::task::957::TaskManager.Task::(_decref) Task=`dc489f29-2c99-4ee1-833c-0b49f2648ea9`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 08:01:39,654::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:01:39,698::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00147689 s, 2.8 MB/s\n'; <rc> = 0
>Thread-15848::DEBUG::2013-03-18 08:01:40,134::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15848::DEBUG::2013-03-18 08:01:40,136::task::568::TaskManager.Task::(_updateState) Task=`c520a254-9d84-4722-923a-bb6b6b1fe6d6`::moving from state init -> state preparing
>Thread-15848::INFO::2013-03-18 08:01:40,136::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15848::INFO::2013-03-18 08:01:40,137::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15848::DEBUG::2013-03-18 08:01:40,137::task::1151::TaskManager.Task::(prepare) Task=`c520a254-9d84-4722-923a-bb6b6b1fe6d6`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15848::DEBUG::2013-03-18 08:01:40,137::task::568::TaskManager.Task::(_updateState) Task=`c520a254-9d84-4722-923a-bb6b6b1fe6d6`::moving from state preparing -> state finished
>Thread-15848::DEBUG::2013-03-18 08:01:40,138::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15848::DEBUG::2013-03-18 08:01:40,138::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15848::DEBUG::2013-03-18 08:01:40,138::task::957::TaskManager.Task::(_decref) Task=`c520a254-9d84-4722-923a-bb6b6b1fe6d6`::ref 0 aborting False
>Thread-15849::DEBUG::2013-03-18 08:01:40,153::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15849::DEBUG::2013-03-18 08:01:40,154::task::568::TaskManager.Task::(_updateState) Task=`22ba57d5-f115-4f05-bcfd-c5cadcb5fd69`::moving from state init -> state preparing
>Thread-15849::INFO::2013-03-18 08:01:40,154::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15849::DEBUG::2013-03-18 08:01:40,155::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`01c77b99-656c-487f-a69e-03a6e6821a57`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15849::DEBUG::2013-03-18 08:01:40,155::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-15849::DEBUG::2013-03-18 08:01:40,156::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15849::DEBUG::2013-03-18 08:01:40,156::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`01c77b99-656c-487f-a69e-03a6e6821a57`::Granted request
>Thread-15849::DEBUG::2013-03-18 08:01:40,157::task::794::TaskManager.Task::(resourceAcquired) Task=`22ba57d5-f115-4f05-bcfd-c5cadcb5fd69`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-15849::DEBUG::2013-03-18 08:01:40,157::task::957::TaskManager.Task::(_decref) Task=`22ba57d5-f115-4f05-bcfd-c5cadcb5fd69`::ref 1 aborting False
>Thread-15849::INFO::2013-03-18 08:01:40,158::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2338324480', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15849::DEBUG::2013-03-18 08:01:40,159::task::1151::TaskManager.Task::(prepare) Task=`22ba57d5-f115-4f05-bcfd-c5cadcb5fd69`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2338324480', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15849::DEBUG::2013-03-18 08:01:40,159::task::568::TaskManager.Task::(_updateState) Task=`22ba57d5-f115-4f05-bcfd-c5cadcb5fd69`::moving from state preparing -> state finished
>Thread-15849::DEBUG::2013-03-18 08:01:40,159::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-15849::DEBUG::2013-03-18 08:01:40,159::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15849::DEBUG::2013-03-18 08:01:40,160::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15849::DEBUG::2013-03-18 08:01:40,160::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15849::DEBUG::2013-03-18 08:01:40,160::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15849::DEBUG::2013-03-18 08:01:40,160::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15849::DEBUG::2013-03-18 08:01:40,161::task::957::TaskManager.Task::(_decref) Task=`22ba57d5-f115-4f05-bcfd-c5cadcb5fd69`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:01:40,860::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:01:41,026::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0662876 s, 15.4 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:01:43,046::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:01:43,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0809042 s, 12.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:01:44,837::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:01:44,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000848035 s, 4.8 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:01:45,247::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:01:45,430::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0825344 s, 12.4 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:01:47,451::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:01:47,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0969115 s, 10.6 MB/s\n'; <rc> = 0
>Thread-15854::DEBUG::2013-03-18 08:01:49,632::task::568::TaskManager.Task::(_updateState) Task=`255bca1f-6f71-4e2f-bf3a-ab7e923ed5e1`::moving from state init -> state preparing
>Thread-15854::INFO::2013-03-18 08:01:49,633::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15854::INFO::2013-03-18 08:01:49,634::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0459630489349', 'lastCheck': '9.9', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0103938579559', 'lastCheck': '1.5', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.03133893013', 'lastCheck': '4.8', 'code': 0, 'valid': True}}
>Thread-15854::DEBUG::2013-03-18 08:01:49,634::task::1151::TaskManager.Task::(prepare) Task=`255bca1f-6f71-4e2f-bf3a-ab7e923ed5e1`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0459630489349', 'lastCheck': '9.9', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0103938579559', 'lastCheck': '1.5', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.03133893013', 'lastCheck': '4.8', 'code': 0, 'valid': True}}
>Thread-15854::DEBUG::2013-03-18 08:01:49,634::task::568::TaskManager.Task::(_updateState) Task=`255bca1f-6f71-4e2f-bf3a-ab7e923ed5e1`::moving from state preparing -> state finished
>Thread-15854::DEBUG::2013-03-18 08:01:49,635::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15854::DEBUG::2013-03-18 08:01:49,635::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15854::DEBUG::2013-03-18 08:01:49,635::task::957::TaskManager.Task::(_decref) Task=`255bca1f-6f71-4e2f-bf3a-ab7e923ed5e1`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:01:49,694::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:01:49,748::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:01:49,787::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00700741 s, 585 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:01:50,175::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.337294 s, 3.0 MB/s\n'; <rc> = 0
>Thread-15856::DEBUG::2013-03-18 08:01:50,269::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15856::DEBUG::2013-03-18 08:01:50,271::task::568::TaskManager.Task::(_updateState) Task=`07ce2938-1186-42ff-a3fa-e1e0a839f90d`::moving from state init -> state preparing
>Thread-15856::INFO::2013-03-18 08:01:50,272::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15856::INFO::2013-03-18 08:01:50,273::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15856::DEBUG::2013-03-18 08:01:50,273::task::1151::TaskManager.Task::(prepare) Task=`07ce2938-1186-42ff-a3fa-e1e0a839f90d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15856::DEBUG::2013-03-18 08:01:50,274::task::568::TaskManager.Task::(_updateState) Task=`07ce2938-1186-42ff-a3fa-e1e0a839f90d`::moving from state preparing -> state finished
>Thread-15856::DEBUG::2013-03-18 08:01:50,274::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15856::DEBUG::2013-03-18 08:01:50,274::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15856::DEBUG::2013-03-18 08:01:50,275::task::957::TaskManager.Task::(_decref) Task=`07ce2938-1186-42ff-a3fa-e1e0a839f90d`::ref 0 aborting False
>Thread-15857::DEBUG::2013-03-18 08:01:50,298::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15857::DEBUG::2013-03-18 08:01:50,299::task::568::TaskManager.Task::(_updateState) Task=`3c9e4c36-7ead-437f-afbf-b54c09397fa9`::moving from state init -> state preparing
>Thread-15857::INFO::2013-03-18 08:01:50,299::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15857::DEBUG::2013-03-18 08:01:50,301::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`4699f82a-48c1-408a-a844-3e6a0b74989c`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15857::DEBUG::2013-03-18 08:01:50,301::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-15857::DEBUG::2013-03-18 08:01:50,302::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15857::DEBUG::2013-03-18 08:01:50,302::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`4699f82a-48c1-408a-a844-3e6a0b74989c`::Granted request
>Thread-15857::DEBUG::2013-03-18 08:01:50,303::task::794::TaskManager.Task::(resourceAcquired) Task=`3c9e4c36-7ead-437f-afbf-b54c09397fa9`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-15857::DEBUG::2013-03-18 08:01:50,303::task::957::TaskManager.Task::(_decref) Task=`3c9e4c36-7ead-437f-afbf-b54c09397fa9`::ref 1 aborting False
>Thread-15857::INFO::2013-03-18 08:01:50,306::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2338324480', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15857::DEBUG::2013-03-18 08:01:50,306::task::1151::TaskManager.Task::(prepare) Task=`3c9e4c36-7ead-437f-afbf-b54c09397fa9`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2338324480', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15857::DEBUG::2013-03-18 08:01:50,307::task::568::TaskManager.Task::(_updateState) Task=`3c9e4c36-7ead-437f-afbf-b54c09397fa9`::moving from state preparing -> state finished
>Thread-15857::DEBUG::2013-03-18 08:01:50,307::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-15857::DEBUG::2013-03-18 08:01:50,307::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15857::DEBUG::2013-03-18 08:01:50,308::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15857::DEBUG::2013-03-18 08:01:50,308::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15857::DEBUG::2013-03-18 08:01:50,309::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15857::DEBUG::2013-03-18 08:01:50,309::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15857::DEBUG::2013-03-18 08:01:50,309::task::957::TaskManager.Task::(_decref) Task=`3c9e4c36-7ead-437f-afbf-b54c09397fa9`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:01:52,194::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:01:52,365::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0720621 s, 14.2 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:01:54,387::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:01:54,599::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0962169 s, 10.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:01:54,873::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:01:54,919::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0184506 s, 222 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:01:56,621::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:01:56,808::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0861135 s, 11.9 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:01:58,828::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:01:59,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0904649 s, 11.3 MB/s\n'; <rc> = 0
>Thread-15862::DEBUG::2013-03-18 08:01:59,966::task::568::TaskManager.Task::(_updateState) Task=`54c2b968-33aa-49d4-8f39-5808234d1ac9`::moving from state init -> state preparing
>Thread-15862::INFO::2013-03-18 08:01:59,967::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15862::INFO::2013-03-18 08:01:59,968::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0433530807495', 'lastCheck': '9.9', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00941514968872', 'lastCheck': '1.8', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0479700565338', 'lastCheck': '5.0', 'code': 0, 'valid': True}}
>Thread-15862::DEBUG::2013-03-18 08:01:59,968::task::1151::TaskManager.Task::(prepare) Task=`54c2b968-33aa-49d4-8f39-5808234d1ac9`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0433530807495', 'lastCheck': '9.9', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00941514968872', 'lastCheck': '1.8', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0479700565338', 'lastCheck': '5.0', 'code': 0, 'valid': True}}
>Thread-15862::DEBUG::2013-03-18 08:01:59,969::task::568::TaskManager.Task::(_updateState) Task=`54c2b968-33aa-49d4-8f39-5808234d1ac9`::moving from state preparing -> state finished
>Thread-15862::DEBUG::2013-03-18 08:01:59,969::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15862::DEBUG::2013-03-18 08:01:59,969::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15862::DEBUG::2013-03-18 08:01:59,969::task::957::TaskManager.Task::(_decref) Task=`54c2b968-33aa-49d4-8f39-5808234d1ac9`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 08:02:00,112::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:02:00,141::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00119145 s, 3.4 MB/s\n'; <rc> = 0
>Thread-15864::DEBUG::2013-03-18 08:02:00,417::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15864::DEBUG::2013-03-18 08:02:00,419::task::568::TaskManager.Task::(_updateState) Task=`3b3b6ad8-bee9-4b5d-b369-21afa1716534`::moving from state init -> state preparing
>Thread-15864::INFO::2013-03-18 08:02:00,420::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15864::INFO::2013-03-18 08:02:00,421::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15864::DEBUG::2013-03-18 08:02:00,422::task::1151::TaskManager.Task::(prepare) Task=`3b3b6ad8-bee9-4b5d-b369-21afa1716534`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15864::DEBUG::2013-03-18 08:02:00,422::task::568::TaskManager.Task::(_updateState) Task=`3b3b6ad8-bee9-4b5d-b369-21afa1716534`::moving from state preparing -> state finished
>Thread-15864::DEBUG::2013-03-18 08:02:00,422::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15864::DEBUG::2013-03-18 08:02:00,423::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15864::DEBUG::2013-03-18 08:02:00,423::task::957::TaskManager.Task::(_decref) Task=`3b3b6ad8-bee9-4b5d-b369-21afa1716534`::ref 0 aborting False
>Thread-15865::DEBUG::2013-03-18 08:02:00,445::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15865::DEBUG::2013-03-18 08:02:00,446::task::568::TaskManager.Task::(_updateState) Task=`6973b111-c6f0-490e-ad87-dce2e70f3ca0`::moving from state init -> state preparing
>Thread-15865::INFO::2013-03-18 08:02:00,446::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15865::DEBUG::2013-03-18 08:02:00,447::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`b2c79a93-769c-4bc9-97b0-553582cf0e25`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15865::DEBUG::2013-03-18 08:02:00,447::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-15865::DEBUG::2013-03-18 08:02:00,448::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15865::DEBUG::2013-03-18 08:02:00,448::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`b2c79a93-769c-4bc9-97b0-553582cf0e25`::Granted request
>Thread-15865::DEBUG::2013-03-18 08:02:00,451::task::794::TaskManager.Task::(resourceAcquired) Task=`6973b111-c6f0-490e-ad87-dce2e70f3ca0`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-15865::DEBUG::2013-03-18 08:02:00,451::task::957::TaskManager.Task::(_decref) Task=`6973b111-c6f0-490e-ad87-dce2e70f3ca0`::ref 1 aborting False
>Thread-15865::INFO::2013-03-18 08:02:00,453::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15865::DEBUG::2013-03-18 08:02:00,454::task::1151::TaskManager.Task::(prepare) Task=`6973b111-c6f0-490e-ad87-dce2e70f3ca0`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15865::DEBUG::2013-03-18 08:02:00,454::task::568::TaskManager.Task::(_updateState) Task=`6973b111-c6f0-490e-ad87-dce2e70f3ca0`::moving from state preparing -> state finished
>Thread-15865::DEBUG::2013-03-18 08:02:00,454::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-15865::DEBUG::2013-03-18 08:02:00,455::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15865::DEBUG::2013-03-18 08:02:00,455::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15865::DEBUG::2013-03-18 08:02:00,455::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15865::DEBUG::2013-03-18 08:02:00,457::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15865::DEBUG::2013-03-18 08:02:00,457::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15865::DEBUG::2013-03-18 08:02:00,458::task::957::TaskManager.Task::(_decref) Task=`6973b111-c6f0-490e-ad87-dce2e70f3ca0`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:02:01,036::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:02:01,228::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0930692 s, 11.0 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:02:03,249::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:02:03,450::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0988439 s, 10.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:02:04,926::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:02:04,974::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.020069 s, 204 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:02:05,474::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:02:05,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0989114 s, 10.4 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:02:07,693::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:02:07,880::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0870498 s, 11.8 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:02:09,916::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:02:10,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0891173 s, 11.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:02:10,191::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-15870::DEBUG::2013-03-18 08:02:10,263::task::568::TaskManager.Task::(_updateState) Task=`a6f15262-6763-4684-9468-0fcf979737e0`::moving from state init -> state preparing
>Thread-15870::INFO::2013-03-18 08:02:10,272::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15870::INFO::2013-03-18 08:02:10,277::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0313770771027', 'lastCheck': '10.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00952291488647', 'lastCheck': '2.0', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.049751996994', 'lastCheck': '5.3', 'code': 0, 'valid': True}}
>Thread-15870::DEBUG::2013-03-18 08:02:10,278::task::1151::TaskManager.Task::(prepare) Task=`a6f15262-6763-4684-9468-0fcf979737e0`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0313770771027', 'lastCheck': '10.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00952291488647', 'lastCheck': '2.0', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.049751996994', 'lastCheck': '5.3', 'code': 0, 'valid': True}}
>Thread-15870::DEBUG::2013-03-18 08:02:10,279::task::568::TaskManager.Task::(_updateState) Task=`a6f15262-6763-4684-9468-0fcf979737e0`::moving from state preparing -> state finished
>Thread-15870::DEBUG::2013-03-18 08:02:10,280::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15870::DEBUG::2013-03-18 08:02:10,281::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-20::DEBUG::2013-03-18 08:02:10,283::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0120486 s, 340 kB/s\n'; <rc> = 0
>Thread-15870::DEBUG::2013-03-18 08:02:10,284::task::957::TaskManager.Task::(_decref) Task=`a6f15262-6763-4684-9468-0fcf979737e0`::ref 0 aborting False
>Thread-15872::DEBUG::2013-03-18 08:02:10,544::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15872::DEBUG::2013-03-18 08:02:10,545::task::568::TaskManager.Task::(_updateState) Task=`54c7cbc5-f301-4f88-8670-e22952198d69`::moving from state init -> state preparing
>Thread-15872::INFO::2013-03-18 08:02:10,545::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15872::INFO::2013-03-18 08:02:10,546::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15872::DEBUG::2013-03-18 08:02:10,546::task::1151::TaskManager.Task::(prepare) Task=`54c7cbc5-f301-4f88-8670-e22952198d69`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15872::DEBUG::2013-03-18 08:02:10,546::task::568::TaskManager.Task::(_updateState) Task=`54c7cbc5-f301-4f88-8670-e22952198d69`::moving from state preparing -> state finished
>Thread-15872::DEBUG::2013-03-18 08:02:10,546::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15872::DEBUG::2013-03-18 08:02:10,547::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15872::DEBUG::2013-03-18 08:02:10,547::task::957::TaskManager.Task::(_decref) Task=`54c7cbc5-f301-4f88-8670-e22952198d69`::ref 0 aborting False
>Thread-15873::DEBUG::2013-03-18 08:02:10,567::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15873::DEBUG::2013-03-18 08:02:10,568::task::568::TaskManager.Task::(_updateState) Task=`65b6479d-e689-43af-a0c0-150968daed1d`::moving from state init -> state preparing
>Thread-15873::INFO::2013-03-18 08:02:10,569::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15873::DEBUG::2013-03-18 08:02:10,570::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`0d56b6e6-dfe0-4166-9fbe-14fa6dfb0b38`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15873::DEBUG::2013-03-18 08:02:10,570::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-15873::DEBUG::2013-03-18 08:02:10,571::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15873::DEBUG::2013-03-18 08:02:10,572::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`0d56b6e6-dfe0-4166-9fbe-14fa6dfb0b38`::Granted request
>Thread-15873::DEBUG::2013-03-18 08:02:10,573::task::794::TaskManager.Task::(resourceAcquired) Task=`65b6479d-e689-43af-a0c0-150968daed1d`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-15873::DEBUG::2013-03-18 08:02:10,573::task::957::TaskManager.Task::(_decref) Task=`65b6479d-e689-43af-a0c0-150968daed1d`::ref 1 aborting False
>Thread-15873::INFO::2013-03-18 08:02:10,575::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15873::DEBUG::2013-03-18 08:02:10,576::task::1151::TaskManager.Task::(prepare) Task=`65b6479d-e689-43af-a0c0-150968daed1d`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15873::DEBUG::2013-03-18 08:02:10,576::task::568::TaskManager.Task::(_updateState) Task=`65b6479d-e689-43af-a0c0-150968daed1d`::moving from state preparing -> state finished
>Thread-15873::DEBUG::2013-03-18 08:02:10,577::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-15873::DEBUG::2013-03-18 08:02:10,577::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15873::DEBUG::2013-03-18 08:02:10,577::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15873::DEBUG::2013-03-18 08:02:10,578::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15873::DEBUG::2013-03-18 08:02:10,578::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15873::DEBUG::2013-03-18 08:02:10,578::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15873::DEBUG::2013-03-18 08:02:10,579::task::957::TaskManager.Task::(_decref) Task=`65b6479d-e689-43af-a0c0-150968daed1d`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:02:12,106::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:02:12,303::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0964989 s, 10.6 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:02:14,323::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:02:14,538::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0775025 s, 13.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:02:14,981::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:02:15,014::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00770367 s, 532 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:02:16,559::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:02:16,752::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0874471 s, 11.7 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:02:18,773::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:02:18,961::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0902902 s, 11.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:02:20,377::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:02:20,404::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000865165 s, 4.7 MB/s\n'; <rc> = 0
>Thread-15878::DEBUG::2013-03-18 08:02:20,663::task::568::TaskManager.Task::(_updateState) Task=`55c99381-23e6-44c1-9294-b195c7477009`::moving from state init -> state preparing
>Thread-15878::INFO::2013-03-18 08:02:20,664::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15878::INFO::2013-03-18 08:02:20,665::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0295119285583', 'lastCheck': '0.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00920915603638', 'lastCheck': '2.4', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0354619026184', 'lastCheck': '5.6', 'code': 0, 'valid': True}}
>Thread-15878::DEBUG::2013-03-18 08:02:20,665::task::1151::TaskManager.Task::(prepare) Task=`55c99381-23e6-44c1-9294-b195c7477009`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0295119285583', 'lastCheck': '0.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00920915603638', 'lastCheck': '2.4', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0354619026184', 'lastCheck': '5.6', 'code': 0, 'valid': True}}
>Thread-15878::DEBUG::2013-03-18 08:02:20,665::task::568::TaskManager.Task::(_updateState) Task=`55c99381-23e6-44c1-9294-b195c7477009`::moving from state preparing -> state finished
>Thread-15878::DEBUG::2013-03-18 08:02:20,666::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15878::DEBUG::2013-03-18 08:02:20,666::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15878::DEBUG::2013-03-18 08:02:20,666::task::957::TaskManager.Task::(_decref) Task=`55c99381-23e6-44c1-9294-b195c7477009`::ref 0 aborting False
>Thread-15879::DEBUG::2013-03-18 08:02:20,693::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15879::DEBUG::2013-03-18 08:02:20,693::task::568::TaskManager.Task::(_updateState) Task=`c2a248f3-e8e5-40bb-bb83-d48dc40c6251`::moving from state init -> state preparing
>Thread-15879::INFO::2013-03-18 08:02:20,694::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15879::INFO::2013-03-18 08:02:20,694::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15879::DEBUG::2013-03-18 08:02:20,694::task::1151::TaskManager.Task::(prepare) Task=`c2a248f3-e8e5-40bb-bb83-d48dc40c6251`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15879::DEBUG::2013-03-18 08:02:20,695::task::568::TaskManager.Task::(_updateState) Task=`c2a248f3-e8e5-40bb-bb83-d48dc40c6251`::moving from state preparing -> state finished
>Thread-15879::DEBUG::2013-03-18 08:02:20,695::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15879::DEBUG::2013-03-18 08:02:20,695::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15879::DEBUG::2013-03-18 08:02:20,695::task::957::TaskManager.Task::(_decref) Task=`c2a248f3-e8e5-40bb-bb83-d48dc40c6251`::ref 0 aborting False
>Thread-15881::DEBUG::2013-03-18 08:02:20,711::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15881::DEBUG::2013-03-18 08:02:20,712::task::568::TaskManager.Task::(_updateState) Task=`04f09436-7cae-4559-aaf1-621a848df01b`::moving from state init -> state preparing
>Thread-15881::INFO::2013-03-18 08:02:20,713::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15881::DEBUG::2013-03-18 08:02:20,714::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`ca681a87-13e3-456c-b996-232306e33915`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15881::DEBUG::2013-03-18 08:02:20,714::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-15881::DEBUG::2013-03-18 08:02:20,715::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15881::DEBUG::2013-03-18 08:02:20,715::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`ca681a87-13e3-456c-b996-232306e33915`::Granted request
>Thread-15881::DEBUG::2013-03-18 08:02:20,715::task::794::TaskManager.Task::(resourceAcquired) Task=`04f09436-7cae-4559-aaf1-621a848df01b`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-15881::DEBUG::2013-03-18 08:02:20,716::task::957::TaskManager.Task::(_decref) Task=`04f09436-7cae-4559-aaf1-621a848df01b`::ref 1 aborting False
>Thread-15881::INFO::2013-03-18 08:02:20,718::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15881::DEBUG::2013-03-18 08:02:20,718::task::1151::TaskManager.Task::(prepare) Task=`04f09436-7cae-4559-aaf1-621a848df01b`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15881::DEBUG::2013-03-18 08:02:20,718::task::568::TaskManager.Task::(_updateState) Task=`04f09436-7cae-4559-aaf1-621a848df01b`::moving from state preparing -> state finished
>Thread-15881::DEBUG::2013-03-18 08:02:20,718::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-15881::DEBUG::2013-03-18 08:02:20,719::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15881::DEBUG::2013-03-18 08:02:20,719::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15881::DEBUG::2013-03-18 08:02:20,719::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15881::DEBUG::2013-03-18 08:02:20,720::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15881::DEBUG::2013-03-18 08:02:20,720::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15881::DEBUG::2013-03-18 08:02:20,720::task::957::TaskManager.Task::(_decref) Task=`04f09436-7cae-4559-aaf1-621a848df01b`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:02:20,983::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:02:21,176::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0843328 s, 12.1 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:02:23,196::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:02:23,370::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0761573 s, 13.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:02:25,022::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:02:25,060::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00165191 s, 2.5 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:02:25,392::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:02:25,568::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0798272 s, 12.8 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:02:27,588::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:02:27,776::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0889521 s, 11.5 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:02:29,797::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:02:29,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0722776 s, 14.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:02:30,453::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:02:30,499::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0117877 s, 347 kB/s\n'; <rc> = 0
>Thread-15886::DEBUG::2013-03-18 08:02:30,856::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15886::DEBUG::2013-03-18 08:02:30,858::task::568::TaskManager.Task::(_updateState) Task=`29167c84-01c8-4037-a7e5-87c89c6d5d5e`::moving from state init -> state preparing
>Thread-15886::INFO::2013-03-18 08:02:30,859::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15886::INFO::2013-03-18 08:02:30,860::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15886::DEBUG::2013-03-18 08:02:30,860::task::1151::TaskManager.Task::(prepare) Task=`29167c84-01c8-4037-a7e5-87c89c6d5d5e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15886::DEBUG::2013-03-18 08:02:30,860::task::568::TaskManager.Task::(_updateState) Task=`29167c84-01c8-4037-a7e5-87c89c6d5d5e`::moving from state preparing -> state finished
>Thread-15886::DEBUG::2013-03-18 08:02:30,860::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15886::DEBUG::2013-03-18 08:02:30,861::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15886::DEBUG::2013-03-18 08:02:30,861::task::957::TaskManager.Task::(_decref) Task=`29167c84-01c8-4037-a7e5-87c89c6d5d5e`::ref 0 aborting False
>Thread-15887::DEBUG::2013-03-18 08:02:30,887::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15887::DEBUG::2013-03-18 08:02:30,888::task::568::TaskManager.Task::(_updateState) Task=`fbe536e9-3996-42a5-88bf-f61fccbecf99`::moving from state init -> state preparing
>Thread-15887::INFO::2013-03-18 08:02:30,888::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15887::DEBUG::2013-03-18 08:02:30,890::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`a57c2bcf-6dd3-47b5-a346-3f9f9d3576ca`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15887::DEBUG::2013-03-18 08:02:30,890::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-15887::DEBUG::2013-03-18 08:02:30,891::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15887::DEBUG::2013-03-18 08:02:30,891::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`a57c2bcf-6dd3-47b5-a346-3f9f9d3576ca`::Granted request
>Thread-15887::DEBUG::2013-03-18 08:02:30,892::task::794::TaskManager.Task::(resourceAcquired) Task=`fbe536e9-3996-42a5-88bf-f61fccbecf99`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-15887::DEBUG::2013-03-18 08:02:30,893::task::957::TaskManager.Task::(_decref) Task=`fbe536e9-3996-42a5-88bf-f61fccbecf99`::ref 1 aborting False
>Thread-15887::INFO::2013-03-18 08:02:30,895::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15887::DEBUG::2013-03-18 08:02:30,896::task::1151::TaskManager.Task::(prepare) Task=`fbe536e9-3996-42a5-88bf-f61fccbecf99`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15887::DEBUG::2013-03-18 08:02:30,896::task::568::TaskManager.Task::(_updateState) Task=`fbe536e9-3996-42a5-88bf-f61fccbecf99`::moving from state preparing -> state finished
>Thread-15887::DEBUG::2013-03-18 08:02:30,897::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-15887::DEBUG::2013-03-18 08:02:30,897::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15887::DEBUG::2013-03-18 08:02:30,897::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15887::DEBUG::2013-03-18 08:02:30,898::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15887::DEBUG::2013-03-18 08:02:30,898::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15887::DEBUG::2013-03-18 08:02:30,899::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15887::DEBUG::2013-03-18 08:02:30,899::task::957::TaskManager.Task::(_decref) Task=`fbe536e9-3996-42a5-88bf-f61fccbecf99`::ref 0 aborting False
>Thread-15888::DEBUG::2013-03-18 08:02:30,953::task::568::TaskManager.Task::(_updateState) Task=`28f19fb5-3fd6-42f6-832e-c516355d61eb`::moving from state init -> state preparing
>Thread-15888::INFO::2013-03-18 08:02:30,954::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15888::INFO::2013-03-18 08:02:30,954::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0484628677368', 'lastCheck': '0.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0113298892975', 'lastCheck': '2.6', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0402269363403', 'lastCheck': '5.9', 'code': 0, 'valid': True}}
>Thread-15888::DEBUG::2013-03-18 08:02:30,955::task::1151::TaskManager.Task::(prepare) Task=`28f19fb5-3fd6-42f6-832e-c516355d61eb`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0484628677368', 'lastCheck': '0.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0113298892975', 'lastCheck': '2.6', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0402269363403', 'lastCheck': '5.9', 'code': 0, 'valid': True}}
>Thread-15888::DEBUG::2013-03-18 08:02:30,955::task::568::TaskManager.Task::(_updateState) Task=`28f19fb5-3fd6-42f6-832e-c516355d61eb`::moving from state preparing -> state finished
>Thread-15888::DEBUG::2013-03-18 08:02:30,955::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15888::DEBUG::2013-03-18 08:02:30,956::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15888::DEBUG::2013-03-18 08:02:30,956::task::957::TaskManager.Task::(_decref) Task=`28f19fb5-3fd6-42f6-832e-c516355d61eb`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:02:32,007::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:02:32,165::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0530386 s, 19.3 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:02:34,186::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:02:34,344::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0610612 s, 16.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:02:35,068::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:02:35,121::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0183624 s, 223 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:02:36,366::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:02:36,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.10845 s, 9.4 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:02:38,602::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:02:38,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0922491 s, 11.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:02:40,546::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:02:40,581::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0080668 s, 508 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:02:40,813::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:02:40,996::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0873203 s, 11.7 MB/s\n'; <rc> = 0
>Thread-15894::DEBUG::2013-03-18 08:02:41,038::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15894::DEBUG::2013-03-18 08:02:41,041::task::568::TaskManager.Task::(_updateState) Task=`b748e638-cc82-4544-aee8-98c9a3047f2e`::moving from state init -> state preparing
>Thread-15894::INFO::2013-03-18 08:02:41,042::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15894::INFO::2013-03-18 08:02:41,043::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15894::DEBUG::2013-03-18 08:02:41,043::task::1151::TaskManager.Task::(prepare) Task=`b748e638-cc82-4544-aee8-98c9a3047f2e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15894::DEBUG::2013-03-18 08:02:41,043::task::568::TaskManager.Task::(_updateState) Task=`b748e638-cc82-4544-aee8-98c9a3047f2e`::moving from state preparing -> state finished
>Thread-15894::DEBUG::2013-03-18 08:02:41,044::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15894::DEBUG::2013-03-18 08:02:41,044::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15894::DEBUG::2013-03-18 08:02:41,044::task::957::TaskManager.Task::(_decref) Task=`b748e638-cc82-4544-aee8-98c9a3047f2e`::ref 0 aborting False
>Thread-15895::DEBUG::2013-03-18 08:02:41,066::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15895::DEBUG::2013-03-18 08:02:41,067::task::568::TaskManager.Task::(_updateState) Task=`f9555ce3-ac51-4f89-9798-a1adf6bd9880`::moving from state init -> state preparing
>Thread-15895::INFO::2013-03-18 08:02:41,068::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15895::DEBUG::2013-03-18 08:02:41,069::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`9c1f12e3-b490-4041-965a-cc2ce7ac5a64`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15895::DEBUG::2013-03-18 08:02:41,070::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-15895::DEBUG::2013-03-18 08:02:41,071::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15895::DEBUG::2013-03-18 08:02:41,071::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`9c1f12e3-b490-4041-965a-cc2ce7ac5a64`::Granted request
>Thread-15895::DEBUG::2013-03-18 08:02:41,072::task::794::TaskManager.Task::(resourceAcquired) Task=`f9555ce3-ac51-4f89-9798-a1adf6bd9880`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-15895::DEBUG::2013-03-18 08:02:41,072::task::957::TaskManager.Task::(_decref) Task=`f9555ce3-ac51-4f89-9798-a1adf6bd9880`::ref 1 aborting False
>Thread-15895::INFO::2013-03-18 08:02:41,077::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15895::DEBUG::2013-03-18 08:02:41,077::task::1151::TaskManager.Task::(prepare) Task=`f9555ce3-ac51-4f89-9798-a1adf6bd9880`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15895::DEBUG::2013-03-18 08:02:41,078::task::568::TaskManager.Task::(_updateState) Task=`f9555ce3-ac51-4f89-9798-a1adf6bd9880`::moving from state preparing -> state finished
>Thread-15895::DEBUG::2013-03-18 08:02:41,078::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-15895::DEBUG::2013-03-18 08:02:41,078::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15895::DEBUG::2013-03-18 08:02:41,079::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15895::DEBUG::2013-03-18 08:02:41,079::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15895::DEBUG::2013-03-18 08:02:41,080::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15895::DEBUG::2013-03-18 08:02:41,080::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15895::DEBUG::2013-03-18 08:02:41,081::task::957::TaskManager.Task::(_decref) Task=`f9555ce3-ac51-4f89-9798-a1adf6bd9880`::ref 0 aborting False
>Thread-15896::DEBUG::2013-03-18 08:02:41,241::task::568::TaskManager.Task::(_updateState) Task=`2c6dbe38-428d-4ff8-adf0-518fcdec11f5`::moving from state init -> state preparing
>Thread-15896::INFO::2013-03-18 08:02:41,242::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15896::INFO::2013-03-18 08:02:41,242::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0379860401154', 'lastCheck': '0.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00883793830872', 'lastCheck': '2.8', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0545561313629', 'lastCheck': '6.1', 'code': 0, 'valid': True}}
>Thread-15896::DEBUG::2013-03-18 08:02:41,242::task::1151::TaskManager.Task::(prepare) Task=`2c6dbe38-428d-4ff8-adf0-518fcdec11f5`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0379860401154', 'lastCheck': '0.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00883793830872', 'lastCheck': '2.8', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0545561313629', 'lastCheck': '6.1', 'code': 0, 'valid': True}}
>Thread-15896::DEBUG::2013-03-18 08:02:41,243::task::568::TaskManager.Task::(_updateState) Task=`2c6dbe38-428d-4ff8-adf0-518fcdec11f5`::moving from state preparing -> state finished
>Thread-15896::DEBUG::2013-03-18 08:02:41,243::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15896::DEBUG::2013-03-18 08:02:41,243::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15896::DEBUG::2013-03-18 08:02:41,244::task::957::TaskManager.Task::(_decref) Task=`2c6dbe38-428d-4ff8-adf0-518fcdec11f5`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:02:43,017::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:02:43,209::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0921288 s, 11.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:02:45,126::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:02:45,169::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0132635 s, 309 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:02:45,232::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:02:45,459::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.103395 s, 9.9 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:02:47,481::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:02:47,664::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0866339 s, 11.8 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:02:49,684::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:02:49,867::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0858576 s, 11.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:02:50,631::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:02:50,673::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0145868 s, 281 kB/s\n'; <rc> = 0
>Thread-15902::DEBUG::2013-03-18 08:02:51,210::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15902::DEBUG::2013-03-18 08:02:51,213::task::568::TaskManager.Task::(_updateState) Task=`9598f237-6d1f-496e-b97b-6248ae907543`::moving from state init -> state preparing
>Thread-15902::INFO::2013-03-18 08:02:51,213::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15902::INFO::2013-03-18 08:02:51,214::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15902::DEBUG::2013-03-18 08:02:51,214::task::1151::TaskManager.Task::(prepare) Task=`9598f237-6d1f-496e-b97b-6248ae907543`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15902::DEBUG::2013-03-18 08:02:51,215::task::568::TaskManager.Task::(_updateState) Task=`9598f237-6d1f-496e-b97b-6248ae907543`::moving from state preparing -> state finished
>Thread-15902::DEBUG::2013-03-18 08:02:51,215::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15902::DEBUG::2013-03-18 08:02:51,215::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15902::DEBUG::2013-03-18 08:02:51,215::task::957::TaskManager.Task::(_decref) Task=`9598f237-6d1f-496e-b97b-6248ae907543`::ref 0 aborting False
>Thread-15903::DEBUG::2013-03-18 08:02:51,241::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15903::DEBUG::2013-03-18 08:02:51,242::task::568::TaskManager.Task::(_updateState) Task=`07a3275e-504d-4c3d-9847-414804ff4f07`::moving from state init -> state preparing
>Thread-15903::INFO::2013-03-18 08:02:51,243::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15903::DEBUG::2013-03-18 08:02:51,244::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`b7ba0031-6a68-406d-be2a-a3de35e49d77`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15903::DEBUG::2013-03-18 08:02:51,244::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-15903::DEBUG::2013-03-18 08:02:51,245::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15903::DEBUG::2013-03-18 08:02:51,246::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`b7ba0031-6a68-406d-be2a-a3de35e49d77`::Granted request
>Thread-15903::DEBUG::2013-03-18 08:02:51,246::task::794::TaskManager.Task::(resourceAcquired) Task=`07a3275e-504d-4c3d-9847-414804ff4f07`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-15903::DEBUG::2013-03-18 08:02:51,247::task::957::TaskManager.Task::(_decref) Task=`07a3275e-504d-4c3d-9847-414804ff4f07`::ref 1 aborting False
>Thread-15903::INFO::2013-03-18 08:02:51,250::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15903::DEBUG::2013-03-18 08:02:51,250::task::1151::TaskManager.Task::(prepare) Task=`07a3275e-504d-4c3d-9847-414804ff4f07`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15903::DEBUG::2013-03-18 08:02:51,251::task::568::TaskManager.Task::(_updateState) Task=`07a3275e-504d-4c3d-9847-414804ff4f07`::moving from state preparing -> state finished
>Thread-15903::DEBUG::2013-03-18 08:02:51,251::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-15903::DEBUG::2013-03-18 08:02:51,251::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15903::DEBUG::2013-03-18 08:02:51,252::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15903::DEBUG::2013-03-18 08:02:51,252::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15903::DEBUG::2013-03-18 08:02:51,253::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15903::DEBUG::2013-03-18 08:02:51,253::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15903::DEBUG::2013-03-18 08:02:51,253::task::957::TaskManager.Task::(_decref) Task=`07a3275e-504d-4c3d-9847-414804ff4f07`::ref 0 aborting False
>Thread-15904::DEBUG::2013-03-18 08:02:51,474::task::568::TaskManager.Task::(_updateState) Task=`4d4d9d6c-c44c-4f64-8b7a-ebccc1017ea7`::moving from state init -> state preparing
>Thread-15904::INFO::2013-03-18 08:02:51,475::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15904::INFO::2013-03-18 08:02:51,475::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0435678958893', 'lastCheck': '0.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00844502449036', 'lastCheck': '3.0', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.045175075531', 'lastCheck': '6.3', 'code': 0, 'valid': True}}
>Thread-15904::DEBUG::2013-03-18 08:02:51,477::task::1151::TaskManager.Task::(prepare) Task=`4d4d9d6c-c44c-4f64-8b7a-ebccc1017ea7`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0435678958893', 'lastCheck': '0.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00844502449036', 'lastCheck': '3.0', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.045175075531', 'lastCheck': '6.3', 'code': 0, 'valid': True}}
>Thread-15904::DEBUG::2013-03-18 08:02:51,477::task::568::TaskManager.Task::(_updateState) Task=`4d4d9d6c-c44c-4f64-8b7a-ebccc1017ea7`::moving from state preparing -> state finished
>Thread-15904::DEBUG::2013-03-18 08:02:51,478::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15904::DEBUG::2013-03-18 08:02:51,478::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15904::DEBUG::2013-03-18 08:02:51,478::task::957::TaskManager.Task::(_decref) Task=`4d4d9d6c-c44c-4f64-8b7a-ebccc1017ea7`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:02:51,889::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:02:52,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.101429 s, 10.1 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:02:54,124::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:02:54,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0736497 s, 13.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:02:55,177::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:02:55,225::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0190424 s, 215 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:02:56,315::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:02:56,519::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0960377 s, 10.7 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:02:58,540::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:02:58,721::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0802218 s, 12.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:03:00,720::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:03:00,742::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:03:00,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00782989 s, 523 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:03:01,148::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.300478 s, 3.4 MB/s\n'; <rc> = 0
>Thread-15910::DEBUG::2013-03-18 08:03:01,363::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15910::DEBUG::2013-03-18 08:03:01,365::task::568::TaskManager.Task::(_updateState) Task=`ba0f0f0a-d9c5-40ee-8bfa-40f9ce79248a`::moving from state init -> state preparing
>Thread-15910::INFO::2013-03-18 08:03:01,366::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15910::INFO::2013-03-18 08:03:01,366::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15910::DEBUG::2013-03-18 08:03:01,367::task::1151::TaskManager.Task::(prepare) Task=`ba0f0f0a-d9c5-40ee-8bfa-40f9ce79248a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15910::DEBUG::2013-03-18 08:03:01,367::task::568::TaskManager.Task::(_updateState) Task=`ba0f0f0a-d9c5-40ee-8bfa-40f9ce79248a`::moving from state preparing -> state finished
>Thread-15910::DEBUG::2013-03-18 08:03:01,367::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15910::DEBUG::2013-03-18 08:03:01,368::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15910::DEBUG::2013-03-18 08:03:01,368::task::957::TaskManager.Task::(_decref) Task=`ba0f0f0a-d9c5-40ee-8bfa-40f9ce79248a`::ref 0 aborting False
>Thread-15911::DEBUG::2013-03-18 08:03:01,390::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15911::DEBUG::2013-03-18 08:03:01,391::task::568::TaskManager.Task::(_updateState) Task=`472898c1-a689-44d1-80bc-926cb29ce9df`::moving from state init -> state preparing
>Thread-15911::INFO::2013-03-18 08:03:01,392::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15911::DEBUG::2013-03-18 08:03:01,393::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`306131cc-4ff5-461b-8d92-509d2d28b619`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15911::DEBUG::2013-03-18 08:03:01,393::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-15911::DEBUG::2013-03-18 08:03:01,394::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15911::DEBUG::2013-03-18 08:03:01,394::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`306131cc-4ff5-461b-8d92-509d2d28b619`::Granted request
>Thread-15911::DEBUG::2013-03-18 08:03:01,395::task::794::TaskManager.Task::(resourceAcquired) Task=`472898c1-a689-44d1-80bc-926cb29ce9df`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-15911::DEBUG::2013-03-18 08:03:01,395::task::957::TaskManager.Task::(_decref) Task=`472898c1-a689-44d1-80bc-926cb29ce9df`::ref 1 aborting False
>Thread-15911::INFO::2013-03-18 08:03:01,398::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15911::DEBUG::2013-03-18 08:03:01,398::task::1151::TaskManager.Task::(prepare) Task=`472898c1-a689-44d1-80bc-926cb29ce9df`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15911::DEBUG::2013-03-18 08:03:01,398::task::568::TaskManager.Task::(_updateState) Task=`472898c1-a689-44d1-80bc-926cb29ce9df`::moving from state preparing -> state finished
>Thread-15911::DEBUG::2013-03-18 08:03:01,398::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-15911::DEBUG::2013-03-18 08:03:01,399::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15911::DEBUG::2013-03-18 08:03:01,399::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15911::DEBUG::2013-03-18 08:03:01,399::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15911::DEBUG::2013-03-18 08:03:01,400::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15911::DEBUG::2013-03-18 08:03:01,400::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15911::DEBUG::2013-03-18 08:03:01,400::task::957::TaskManager.Task::(_decref) Task=`472898c1-a689-44d1-80bc-926cb29ce9df`::ref 0 aborting False
>Thread-15912::DEBUG::2013-03-18 08:03:01,757::task::568::TaskManager.Task::(_updateState) Task=`5dd174a9-1d21-49f4-90a7-db5069502398`::moving from state init -> state preparing
>Thread-15912::INFO::2013-03-18 08:03:01,758::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15912::INFO::2013-03-18 08:03:01,758::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0741410255432', 'lastCheck': '0.7', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0101199150085', 'lastCheck': '3.2', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0499169826508', 'lastCheck': '6.5', 'code': 0, 'valid': True}}
>Thread-15912::DEBUG::2013-03-18 08:03:01,759::task::1151::TaskManager.Task::(prepare) Task=`5dd174a9-1d21-49f4-90a7-db5069502398`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0741410255432', 'lastCheck': '0.7', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0101199150085', 'lastCheck': '3.2', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0499169826508', 'lastCheck': '6.5', 'code': 0, 'valid': True}}
>Thread-15912::DEBUG::2013-03-18 08:03:01,759::task::568::TaskManager.Task::(_updateState) Task=`5dd174a9-1d21-49f4-90a7-db5069502398`::moving from state preparing -> state finished
>Thread-15912::DEBUG::2013-03-18 08:03:01,759::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15912::DEBUG::2013-03-18 08:03:01,759::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15912::DEBUG::2013-03-18 08:03:01,760::task::957::TaskManager.Task::(_decref) Task=`5dd174a9-1d21-49f4-90a7-db5069502398`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:03:03,159::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:03:03,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0827339 s, 12.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:03:05,231::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:03:05,267::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00610282 s, 671 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:03:05,358::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:03:05,567::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.089775 s, 11.4 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:03:07,588::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:03:07,773::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0814745 s, 12.6 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:03:09,793::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:03:09,986::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0918356 s, 11.2 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:03:11,079::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:03:11,118::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0122088 s, 335 kB/s\n'; <rc> = 0
>Thread-15918::DEBUG::2013-03-18 08:03:11,519::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15918::DEBUG::2013-03-18 08:03:11,522::task::568::TaskManager.Task::(_updateState) Task=`47d079c9-12ad-4c3f-80d9-2c62fac9d07b`::moving from state init -> state preparing
>Thread-15918::INFO::2013-03-18 08:03:11,523::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15918::INFO::2013-03-18 08:03:11,523::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15918::DEBUG::2013-03-18 08:03:11,524::task::1151::TaskManager.Task::(prepare) Task=`47d079c9-12ad-4c3f-80d9-2c62fac9d07b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15918::DEBUG::2013-03-18 08:03:11,524::task::568::TaskManager.Task::(_updateState) Task=`47d079c9-12ad-4c3f-80d9-2c62fac9d07b`::moving from state preparing -> state finished
>Thread-15918::DEBUG::2013-03-18 08:03:11,524::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15918::DEBUG::2013-03-18 08:03:11,524::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15918::DEBUG::2013-03-18 08:03:11,525::task::957::TaskManager.Task::(_decref) Task=`47d079c9-12ad-4c3f-80d9-2c62fac9d07b`::ref 0 aborting False
>Thread-15919::DEBUG::2013-03-18 08:03:11,546::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15919::DEBUG::2013-03-18 08:03:11,547::task::568::TaskManager.Task::(_updateState) Task=`cf5039ed-21e7-4f02-a5ff-372851842e66`::moving from state init -> state preparing
>Thread-15919::INFO::2013-03-18 08:03:11,547::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15919::DEBUG::2013-03-18 08:03:11,549::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`3bcf83f7-b4be-453d-aa28-795a0bddd4a9`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15919::DEBUG::2013-03-18 08:03:11,549::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-15919::DEBUG::2013-03-18 08:03:11,550::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15919::DEBUG::2013-03-18 08:03:11,550::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`3bcf83f7-b4be-453d-aa28-795a0bddd4a9`::Granted request
>Thread-15919::DEBUG::2013-03-18 08:03:11,551::task::794::TaskManager.Task::(resourceAcquired) Task=`cf5039ed-21e7-4f02-a5ff-372851842e66`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-15919::DEBUG::2013-03-18 08:03:11,552::task::957::TaskManager.Task::(_decref) Task=`cf5039ed-21e7-4f02-a5ff-372851842e66`::ref 1 aborting False
>Thread-15919::INFO::2013-03-18 08:03:11,555::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15919::DEBUG::2013-03-18 08:03:11,555::task::1151::TaskManager.Task::(prepare) Task=`cf5039ed-21e7-4f02-a5ff-372851842e66`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15919::DEBUG::2013-03-18 08:03:11,555::task::568::TaskManager.Task::(_updateState) Task=`cf5039ed-21e7-4f02-a5ff-372851842e66`::moving from state preparing -> state finished
>Thread-15919::DEBUG::2013-03-18 08:03:11,556::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-15919::DEBUG::2013-03-18 08:03:11,556::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15919::DEBUG::2013-03-18 08:03:11,557::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15919::DEBUG::2013-03-18 08:03:11,557::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15919::DEBUG::2013-03-18 08:03:11,557::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15919::DEBUG::2013-03-18 08:03:11,558::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15919::DEBUG::2013-03-18 08:03:11,558::task::957::TaskManager.Task::(_decref) Task=`cf5039ed-21e7-4f02-a5ff-372851842e66`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:03:12,008::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Thread-15920::DEBUG::2013-03-18 08:03:12,136::task::568::TaskManager.Task::(_updateState) Task=`1a346007-d759-4718-9ec7-3c8c54a1741a`::moving from state init -> state preparing
>Thread-15920::INFO::2013-03-18 08:03:12,145::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15920::INFO::2013-03-18 08:03:12,149::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0419888496399', 'lastCheck': '1.0', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0100090503693', 'lastCheck': '3.6', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0378019809723', 'lastCheck': '6.9', 'code': 0, 'valid': True}}
>Thread-15920::DEBUG::2013-03-18 08:03:12,150::task::1151::TaskManager.Task::(prepare) Task=`1a346007-d759-4718-9ec7-3c8c54a1741a`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0419888496399', 'lastCheck': '1.0', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0100090503693', 'lastCheck': '3.6', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0378019809723', 'lastCheck': '6.9', 'code': 0, 'valid': True}}
>Thread-15920::DEBUG::2013-03-18 08:03:12,151::task::568::TaskManager.Task::(_updateState) Task=`1a346007-d759-4718-9ec7-3c8c54a1741a`::moving from state preparing -> state finished
>Thread-15920::DEBUG::2013-03-18 08:03:12,153::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15920::DEBUG::2013-03-18 08:03:12,154::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15920::DEBUG::2013-03-18 08:03:12,156::task::957::TaskManager.Task::(_decref) Task=`1a346007-d759-4718-9ec7-3c8c54a1741a`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:03:12,296::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.191127 s, 5.4 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:03:14,315::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:03:14,494::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0791994 s, 12.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:03:15,274::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:03:15,314::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0125063 s, 328 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:03:16,516::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:03:16,703::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0826989 s, 12.4 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:03:18,723::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:03:18,909::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0890073 s, 11.5 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:03:20,930::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:03:21,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0876551 s, 11.7 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:03:21,168::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:03:21,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00584618 s, 701 kB/s\n'; <rc> = 0
>Thread-15926::DEBUG::2013-03-18 08:03:21,677::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15926::DEBUG::2013-03-18 08:03:21,680::task::568::TaskManager.Task::(_updateState) Task=`d19937d1-aa30-4c78-a0da-5c4cfa62ae7e`::moving from state init -> state preparing
>Thread-15926::INFO::2013-03-18 08:03:21,681::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15926::INFO::2013-03-18 08:03:21,682::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15926::DEBUG::2013-03-18 08:03:21,682::task::1151::TaskManager.Task::(prepare) Task=`d19937d1-aa30-4c78-a0da-5c4cfa62ae7e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15926::DEBUG::2013-03-18 08:03:21,682::task::568::TaskManager.Task::(_updateState) Task=`d19937d1-aa30-4c78-a0da-5c4cfa62ae7e`::moving from state preparing -> state finished
>Thread-15926::DEBUG::2013-03-18 08:03:21,683::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15926::DEBUG::2013-03-18 08:03:21,683::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15926::DEBUG::2013-03-18 08:03:21,683::task::957::TaskManager.Task::(_decref) Task=`d19937d1-aa30-4c78-a0da-5c4cfa62ae7e`::ref 0 aborting False
>Thread-15927::DEBUG::2013-03-18 08:03:21,705::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15927::DEBUG::2013-03-18 08:03:21,705::task::568::TaskManager.Task::(_updateState) Task=`ea14bcc3-624b-4cd0-8370-41162e5a1fe7`::moving from state init -> state preparing
>Thread-15927::INFO::2013-03-18 08:03:21,706::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15927::DEBUG::2013-03-18 08:03:21,707::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`1f2e4ae2-901e-48e9-a85a-0d2341f705a9`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15927::DEBUG::2013-03-18 08:03:21,707::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-15927::DEBUG::2013-03-18 08:03:21,708::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15927::DEBUG::2013-03-18 08:03:21,708::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`1f2e4ae2-901e-48e9-a85a-0d2341f705a9`::Granted request
>Thread-15927::DEBUG::2013-03-18 08:03:21,709::task::794::TaskManager.Task::(resourceAcquired) Task=`ea14bcc3-624b-4cd0-8370-41162e5a1fe7`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-15927::DEBUG::2013-03-18 08:03:21,709::task::957::TaskManager.Task::(_decref) Task=`ea14bcc3-624b-4cd0-8370-41162e5a1fe7`::ref 1 aborting False
>Thread-15927::INFO::2013-03-18 08:03:21,711::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15927::DEBUG::2013-03-18 08:03:21,711::task::1151::TaskManager.Task::(prepare) Task=`ea14bcc3-624b-4cd0-8370-41162e5a1fe7`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15927::DEBUG::2013-03-18 08:03:21,712::task::568::TaskManager.Task::(_updateState) Task=`ea14bcc3-624b-4cd0-8370-41162e5a1fe7`::moving from state preparing -> state finished
>Thread-15927::DEBUG::2013-03-18 08:03:21,712::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-15927::DEBUG::2013-03-18 08:03:21,712::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15927::DEBUG::2013-03-18 08:03:21,713::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15927::DEBUG::2013-03-18 08:03:21,713::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15927::DEBUG::2013-03-18 08:03:21,713::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15927::DEBUG::2013-03-18 08:03:21,713::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15927::DEBUG::2013-03-18 08:03:21,714::task::957::TaskManager.Task::(_decref) Task=`ea14bcc3-624b-4cd0-8370-41162e5a1fe7`::ref 0 aborting False
>Thread-15928::DEBUG::2013-03-18 08:03:22,360::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15928::DEBUG::2013-03-18 08:03:22,360::task::568::TaskManager.Task::(_updateState) Task=`c57a71aa-b52f-426e-9973-c15a2a55249f`::moving from state init -> state preparing
>Thread-15928::INFO::2013-03-18 08:03:22,361::logUtils::37::dispatcher::(wrapper) Run and protect: getDeviceList(storageType=3, options={})
>Thread-15928::DEBUG::2013-03-18 08:03:22,361::misc::1025::SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage)
>Thread-15928::DEBUG::2013-03-18 08:03:22,362::misc::1027::SamplingMethod::(__call__) Got in to sampling method
>Thread-15928::DEBUG::2013-03-18 08:03:22,362::misc::1025::SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan)
>Thread-15928::DEBUG::2013-03-18 08:03:22,362::misc::1027::SamplingMethod::(__call__) Got in to sampling method
>Thread-15928::DEBUG::2013-03-18 08:03:22,363::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/iscsiadm -m session -R' (cwd None)
>Thread-15928::DEBUG::2013-03-18 08:03:22,421::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-15928::DEBUG::2013-03-18 08:03:22,423::misc::1035::SamplingMethod::(__call__) Returning last result
>MainProcess|Thread-15928::DEBUG::2013-03-18 08:03:22,433::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd of=/sys/class/scsi_host/host0/scan' (cwd None)
>Thread-15929::DEBUG::2013-03-18 08:03:22,500::task::568::TaskManager.Task::(_updateState) Task=`1ab753a6-3ed0-420d-8c8d-819fd9ecfcbf`::moving from state init -> state preparing
>Thread-15929::INFO::2013-03-18 08:03:22,501::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15929::INFO::2013-03-18 08:03:22,501::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0336790084839', 'lastCheck': '1.3', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00867700576782', 'lastCheck': '3.9', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0431411266327', 'lastCheck': '7.2', 'code': 0, 'valid': True}}
>Thread-15929::DEBUG::2013-03-18 08:03:22,502::task::1151::TaskManager.Task::(prepare) Task=`1ab753a6-3ed0-420d-8c8d-819fd9ecfcbf`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0336790084839', 'lastCheck': '1.3', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00867700576782', 'lastCheck': '3.9', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0431411266327', 'lastCheck': '7.2', 'code': 0, 'valid': True}}
>Thread-15929::DEBUG::2013-03-18 08:03:22,502::task::568::TaskManager.Task::(_updateState) Task=`1ab753a6-3ed0-420d-8c8d-819fd9ecfcbf`::moving from state preparing -> state finished
>Thread-15929::DEBUG::2013-03-18 08:03:22,502::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15929::DEBUG::2013-03-18 08:03:22,503::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15929::DEBUG::2013-03-18 08:03:22,503::task::957::TaskManager.Task::(_decref) Task=`1ab753a6-3ed0-420d-8c8d-819fd9ecfcbf`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:03:23,134::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:03:23,315::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0854794 s, 12.0 MB/s\n'; <rc> = 0
>MainProcess|Thread-15928::DEBUG::2013-03-18 08:03:23,447::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd of=/sys/class/scsi_host/host1/scan' (cwd None)
>MainProcess|Thread-15928::DEBUG::2013-03-18 08:03:23,460::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd of=/sys/class/scsi_host/host2/scan' (cwd None)
>MainProcess|Thread-15928::DEBUG::2013-03-18 08:03:23,475::iscsi::388::Storage.ISCSI::(forceIScsiScan) Performing SCSI scan, this will take up to 30 seconds
>Thread-22::DEBUG::2013-03-18 08:03:25,322::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:03:25,355::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:03:25,399::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0175305 s, 234 kB/s\n'; <rc> = 0
>Thread-15928::DEBUG::2013-03-18 08:03:25,484::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/multipath' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:03:25,638::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.141954 s, 7.2 MB/s\n'; <rc> = 0
>Thread-15928::DEBUG::2013-03-18 08:03:25,666::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-15928::DEBUG::2013-03-18 08:03:25,668::lvm::482::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-15928::DEBUG::2013-03-18 08:03:25,668::lvm::484::OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-15928::DEBUG::2013-03-18 08:03:25,669::lvm::493::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-15928::DEBUG::2013-03-18 08:03:25,669::lvm::495::OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-15928::DEBUG::2013-03-18 08:03:25,670::lvm::513::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-15928::DEBUG::2013-03-18 08:03:25,670::lvm::515::OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-15928::DEBUG::2013-03-18 08:03:25,670::misc::1035::SamplingMethod::(__call__) Returning last result
>Thread-15928::DEBUG::2013-03-18 08:03:25,671::lvm::339::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' got the operation mutex
>Thread-15928::DEBUG::2013-03-18 08:03:25,677::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm pvs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,size,vg_name,vg_uuid,pe_start,pe_count,pe_alloc_count,mda_count,dev_size' (cwd None)
>Thread-15928::DEBUG::2013-03-18 08:03:26,204::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-15928::DEBUG::2013-03-18 08:03:26,205::lvm::364::OperationMutex::(_reloadpvs) Operation 'lvm reload operation' released the operation mutex
>MainProcess|Thread-15928::DEBUG::2013-03-18 08:03:26,208::misc::83::Storage.Misc.excCmd::(<lambda>) '/sbin/dmsetup status' (cwd None)
>MainProcess|Thread-15928::DEBUG::2013-03-18 08:03:26,230::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>MainProcess|Thread-15928::DEBUG::2013-03-18 08:03:26,238::misc::83::Storage.Misc.excCmd::(<lambda>) '/sbin/scsi_id --page=0x80 --whitelisted --export --replace-whitespace --device=/dev/dm-0' (cwd None)
>MainProcess|Thread-15928::DEBUG::2013-03-18 08:03:26,250::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>MainProcess|Thread-15928::DEBUG::2013-03-18 08:03:26,263::misc::83::Storage.Misc.excCmd::(<lambda>) '/sbin/scsi_id --page=0x80 --whitelisted --export --replace-whitespace --device=/dev/dm-10' (cwd None)
>MainProcess|Thread-15928::DEBUG::2013-03-18 08:03:26,276::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>MainProcess|Thread-15928::DEBUG::2013-03-18 08:03:26,284::misc::83::Storage.Misc.excCmd::(<lambda>) '/sbin/scsi_id --page=0x80 --whitelisted --export --replace-whitespace --device=/dev/dm-11' (cwd None)
>MainProcess|Thread-15928::DEBUG::2013-03-18 08:03:26,295::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>MainProcess|Thread-15928::DEBUG::2013-03-18 08:03:26,301::misc::83::Storage.Misc.excCmd::(<lambda>) '/sbin/scsi_id --page=0x80 --whitelisted --export --replace-whitespace --device=/dev/dm-25' (cwd None)
>MainProcess|Thread-15928::DEBUG::2013-03-18 08:03:26,313::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = ''; <rc> = 1
>Thread-15928::DEBUG::2013-03-18 08:03:26,318::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm pvcreate --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --test --metadatasize 128m --metadatacopies 2 --metadataignore y /dev/mapper/3600140558f280d707b243749b07609a3 /dev/mapper/36001405e2fe1e87706442878688ebb17 /dev/mapper/36001405398701597f844f4eb099342d0' (cwd None)
>Thread-15928::DEBUG::2013-03-18 08:03:26,875::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  TEST MODE: Metadata will NOT be updated and volumes will not be (de)activated.\n  Can\'t initialize physical volume "/dev/mapper/3600140558f280d707b243749b07609a3" of volume group "4bf968ef-020a-47ee-a1ae-f85eaeb8751a" without -ff\n  Can\'t initialize physical volume "/dev/mapper/36001405e2fe1e87706442878688ebb17" of volume group "f33e50cc-eb0a-4720-857d-bc7001a72567" without -ff\n'; <rc> = 5
>Thread-15928::DEBUG::2013-03-18 08:03:26,881::lvm::809::Storage.LVM::(testPVCreate) rc: 5, out: ['  Physical volume "/dev/mapper/36001405398701597f844f4eb099342d0" successfully created'], err: ['  TEST MODE: Metadata will NOT be updated and volumes will not be (de)activated.', '  Can\'t initialize physical volume "/dev/mapper/3600140558f280d707b243749b07609a3" of volume group "4bf968ef-020a-47ee-a1ae-f85eaeb8751a" without -ff', '  Can\'t initialize physical volume "/dev/mapper/36001405e2fe1e87706442878688ebb17" of volume group "f33e50cc-eb0a-4720-857d-bc7001a72567" without -ff'], unusedDevs: set(['/dev/mapper/36001405398701597f844f4eb099342d0']), usedDevs: set(['/dev/mapper/36001405e2fe1e87706442878688ebb17', '/dev/mapper/3600140558f280d707b243749b07609a3'])
>Thread-15928::INFO::2013-03-18 08:03:26,883::logUtils::39::dispatcher::(wrapper) Run and protect: getDeviceList, Return response: {'devList': [{'status': 'used', 'vendorID': 'LIO-ORG', 'capacity': '10737418240', 'fwrev': '4.0', 'vgUUID': 'AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'pathlist': [{'initiatorname': 'default', 'connection': '10.34.1.214', 'iqn': 'iqn.2003-01.org.linux-iscsi.dhcp-1-214.x8664:sn.113e745f4931', 'portal': '1', 'password': '******', 'port': '3260'}], 'logicalblocksize': '512', 'pathstatus': [{'physdev': 'sdc', 'type': 'iSCSI', 'state': 'active', 'lun': '1'}], 'devtype': 'iSCSI', 'physicalblocksize': '512', 'pvUUID': 'APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR', 'serial': 'SLIO-ORGIBLOCK', 'GUID': '3600140558f280d707b243749b07609a3', 'productID': 'IBLOCK'}, {'status': 'used', 'vendorID': 'LIO-ORG', 'capacity': '53687091200', 'fwrev': '4.0', 'vgUUID': '1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'pathlist': [{'initiatorname': 'default', 'connection': '10.34.1.214', 'iqn': 'iqn.2003-01.org.linux-iscsi.dhcp-1-214.x8664:sn.113e745f4931', 'portal': '1', 'password': '******', 'port': '3260'}], 'logicalblocksize': '512', 'pathstatus': [{'physdev': 'sdb', 'type': 'iSCSI', 'state': 'active', 'lun': '0'}], 'devtype': 'iSCSI', 'physicalblocksize': '512', 'pvUUID': '8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T', 'serial': 'SLIO-ORGIBLOCK', 'GUID': '36001405e2fe1e87706442878688ebb17', 'productID': 'IBLOCK'}, {'status': 'free', 'vendorID': 'LIO-ORG', 'capacity': '5368709120', 'fwrev': '4.0', 'vgUUID': '', 'pathlist': [{'initiatorname': 'default', 'connection': '10.34.1.214', 'iqn': 'iqn.2003-01.org.linux-iscsi.dhcp-1-214.x8664:sn.113e745f4931', 'portal': '1', 'password': '******', 'port': '3260'}], 'logicalblocksize': '512', 'pathstatus': [{'physdev': 'sdd', 'type': 'iSCSI', 'state': 'active', 'lun': '2'}], 'devtype': 'iSCSI', 'physicalblocksize': '512', 'pvUUID': '', 'serial': '', 'GUID': '36001405398701597f844f4eb099342d0', 'productID': 'IBLOCK'}]}
>Thread-15928::DEBUG::2013-03-18 08:03:26,883::task::1151::TaskManager.Task::(prepare) Task=`c57a71aa-b52f-426e-9973-c15a2a55249f`::finished: {'devList': [{'status': 'used', 'fwrev': '4.0', 'vgUUID': 'AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'pathlist': [{'connection': '10.34.1.214', 'iqn': 'iqn.2003-01.org.linux-iscsi.dhcp-1-214.x8664:sn.113e745f4931', 'portal': '1', 'port': '3260', 'initiatorname': 'default'}], 'logicalblocksize': '512', 'devtype': 'iSCSI', 'physicalblocksize': '512', 'serial': 'SLIO-ORGIBLOCK', 'GUID': '3600140558f280d707b243749b07609a3', 'productID': 'IBLOCK', 'vendorID': 'LIO-ORG', 'capacity': '10737418240', 'pathstatus': [{'physdev': 'sdc', 'state': 'active', 'type': 'iSCSI', 'lun': '1'}], 'pvUUID': 'APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR'}, {'status': 'used', 'fwrev': '4.0', 'vgUUID': '1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'pathlist': [{'connection': '10.34.1.214', 'iqn': 'iqn.2003-01.org.linux-iscsi.dhcp-1-214.x8664:sn.113e745f4931', 'portal': '1', 'port': '3260', 'initiatorname': 'default'}], 'logicalblocksize': '512', 'devtype': 'iSCSI', 'physicalblocksize': '512', 'serial': 'SLIO-ORGIBLOCK', 'GUID': '36001405e2fe1e87706442878688ebb17', 'productID': 'IBLOCK', 'vendorID': 'LIO-ORG', 'capacity': '53687091200', 'pathstatus': [{'physdev': 'sdb', 'state': 'active', 'type': 'iSCSI', 'lun': '0'}], 'pvUUID': '8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T'}, {'status': 'free', 'fwrev': '4.0', 'vgUUID': '', 'pathlist': [{'connection': '10.34.1.214', 'iqn': 'iqn.2003-01.org.linux-iscsi.dhcp-1-214.x8664:sn.113e745f4931', 'portal': '1', 'port': '3260', 'initiatorname': 'default'}], 'logicalblocksize': '512', 'devtype': 'iSCSI', 'physicalblocksize': '512', 'serial': '', 'GUID': '36001405398701597f844f4eb099342d0', 'productID': 'IBLOCK', 'vendorID': 'LIO-ORG', 'capacity': '5368709120', 'pathstatus': [{'physdev': 'sdd', 'state': 'active', 'type': 'iSCSI', 'lun': '2'}], 'pvUUID': ''}]}
>Thread-15928::DEBUG::2013-03-18 08:03:26,884::task::568::TaskManager.Task::(_updateState) Task=`c57a71aa-b52f-426e-9973-c15a2a55249f`::moving from state preparing -> state finished
>Thread-15928::DEBUG::2013-03-18 08:03:26,884::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15928::DEBUG::2013-03-18 08:03:26,885::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15928::DEBUG::2013-03-18 08:03:26,885::task::957::TaskManager.Task::(_decref) Task=`c57a71aa-b52f-426e-9973-c15a2a55249f`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:03:27,661::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:03:27,845::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0895952 s, 11.4 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:03:29,866::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:03:30,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0716825 s, 14.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:03:31,247::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 08:03:31,248::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:03:31,708::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:03:31,712::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 08:03:31,713::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:03:31,747::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00945812 s, 433 kB/s\n'; <rc> = 0
>Thread-15935::DEBUG::2013-03-18 08:03:31,854::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15935::DEBUG::2013-03-18 08:03:31,857::task::568::TaskManager.Task::(_updateState) Task=`806d3578-8ed9-44bd-b276-31118dda0503`::moving from state init -> state preparing
>Thread-15935::INFO::2013-03-18 08:03:31,858::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15935::INFO::2013-03-18 08:03:31,859::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15935::DEBUG::2013-03-18 08:03:31,859::task::1151::TaskManager.Task::(prepare) Task=`806d3578-8ed9-44bd-b276-31118dda0503`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15935::DEBUG::2013-03-18 08:03:31,859::task::568::TaskManager.Task::(_updateState) Task=`806d3578-8ed9-44bd-b276-31118dda0503`::moving from state preparing -> state finished
>Thread-15935::DEBUG::2013-03-18 08:03:31,860::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15935::DEBUG::2013-03-18 08:03:31,860::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15935::DEBUG::2013-03-18 08:03:31,860::task::957::TaskManager.Task::(_decref) Task=`806d3578-8ed9-44bd-b276-31118dda0503`::ref 0 aborting False
>Thread-15936::DEBUG::2013-03-18 08:03:31,883::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15936::DEBUG::2013-03-18 08:03:31,884::task::568::TaskManager.Task::(_updateState) Task=`d69327d5-817b-463e-941e-d948020777a5`::moving from state init -> state preparing
>Thread-15936::INFO::2013-03-18 08:03:31,886::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15936::DEBUG::2013-03-18 08:03:31,887::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`77e10afb-daa4-4055-bdcc-7e7459223073`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15936::DEBUG::2013-03-18 08:03:31,887::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-15936::DEBUG::2013-03-18 08:03:31,888::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15936::DEBUG::2013-03-18 08:03:31,889::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`77e10afb-daa4-4055-bdcc-7e7459223073`::Granted request
>Thread-15936::DEBUG::2013-03-18 08:03:31,890::task::794::TaskManager.Task::(resourceAcquired) Task=`d69327d5-817b-463e-941e-d948020777a5`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-15936::DEBUG::2013-03-18 08:03:31,890::task::957::TaskManager.Task::(_decref) Task=`d69327d5-817b-463e-941e-d948020777a5`::ref 1 aborting False
>Thread-15936::INFO::2013-03-18 08:03:31,893::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15936::DEBUG::2013-03-18 08:03:31,893::task::1151::TaskManager.Task::(prepare) Task=`d69327d5-817b-463e-941e-d948020777a5`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15936::DEBUG::2013-03-18 08:03:31,894::task::568::TaskManager.Task::(_updateState) Task=`d69327d5-817b-463e-941e-d948020777a5`::moving from state preparing -> state finished
>Thread-15936::DEBUG::2013-03-18 08:03:31,894::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-15936::DEBUG::2013-03-18 08:03:31,894::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15936::DEBUG::2013-03-18 08:03:31,895::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15936::DEBUG::2013-03-18 08:03:31,895::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15936::DEBUG::2013-03-18 08:03:31,896::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15936::DEBUG::2013-03-18 08:03:31,896::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15936::DEBUG::2013-03-18 08:03:31,896::task::957::TaskManager.Task::(_decref) Task=`d69327d5-817b-463e-941e-d948020777a5`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:03:32,057::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:03:32,255::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0938208 s, 10.9 MB/s\n'; <rc> = 0
>Thread-15937::DEBUG::2013-03-18 08:03:32,767::task::568::TaskManager.Task::(_updateState) Task=`a6560ad3-6199-4877-abab-24acdddc50c5`::moving from state init -> state preparing
>Thread-15937::INFO::2013-03-18 08:03:32,768::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15937::INFO::2013-03-18 08:03:32,769::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0361299514771', 'lastCheck': '1.0', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.011538028717', 'lastCheck': '4.1', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0794520378113', 'lastCheck': '7.4', 'code': 0, 'valid': True}}
>Thread-15937::DEBUG::2013-03-18 08:03:32,770::task::1151::TaskManager.Task::(prepare) Task=`a6560ad3-6199-4877-abab-24acdddc50c5`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0361299514771', 'lastCheck': '1.0', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.011538028717', 'lastCheck': '4.1', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0794520378113', 'lastCheck': '7.4', 'code': 0, 'valid': True}}
>Thread-15937::DEBUG::2013-03-18 08:03:32,770::task::568::TaskManager.Task::(_updateState) Task=`a6560ad3-6199-4877-abab-24acdddc50c5`::moving from state preparing -> state finished
>Thread-15937::DEBUG::2013-03-18 08:03:32,770::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15937::DEBUG::2013-03-18 08:03:32,771::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15937::DEBUG::2013-03-18 08:03:32,771::task::957::TaskManager.Task::(_decref) Task=`a6560ad3-6199-4877-abab-24acdddc50c5`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:03:34,276::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:03:34,452::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0766086 s, 13.4 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:03:35,407::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 08:03:35,408::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:03:35,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:03:35,863::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 08:03:35,864::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:03:35,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0206061 s, 199 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:03:36,474::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:03:36,665::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0870828 s, 11.8 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:03:38,685::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:03:38,900::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.106236 s, 9.6 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:03:40,921::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:03:41,134::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.116614 s, 8.8 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:03:41,798::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:03:41,844::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00694839 s, 589 kB/s\n'; <rc> = 0
>Thread-15943::DEBUG::2013-03-18 08:03:42,014::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15943::DEBUG::2013-03-18 08:03:42,021::task::568::TaskManager.Task::(_updateState) Task=`4c3b47ae-0f42-45c7-8926-3e83f74c7b36`::moving from state init -> state preparing
>Thread-15943::INFO::2013-03-18 08:03:42,022::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15943::INFO::2013-03-18 08:03:42,023::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15943::DEBUG::2013-03-18 08:03:42,023::task::1151::TaskManager.Task::(prepare) Task=`4c3b47ae-0f42-45c7-8926-3e83f74c7b36`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15943::DEBUG::2013-03-18 08:03:42,024::task::568::TaskManager.Task::(_updateState) Task=`4c3b47ae-0f42-45c7-8926-3e83f74c7b36`::moving from state preparing -> state finished
>Thread-15943::DEBUG::2013-03-18 08:03:42,024::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15943::DEBUG::2013-03-18 08:03:42,024::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15943::DEBUG::2013-03-18 08:03:42,024::task::957::TaskManager.Task::(_decref) Task=`4c3b47ae-0f42-45c7-8926-3e83f74c7b36`::ref 0 aborting False
>Thread-15944::DEBUG::2013-03-18 08:03:42,047::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15944::DEBUG::2013-03-18 08:03:42,048::task::568::TaskManager.Task::(_updateState) Task=`ce2863e0-8a72-45ac-880b-73b5dc9225c9`::moving from state init -> state preparing
>Thread-15944::INFO::2013-03-18 08:03:42,049::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15944::DEBUG::2013-03-18 08:03:42,050::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`8bf12840-75b2-4c53-836e-7d55b4b62efd`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15944::DEBUG::2013-03-18 08:03:42,050::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-15944::DEBUG::2013-03-18 08:03:42,051::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15944::DEBUG::2013-03-18 08:03:42,052::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`8bf12840-75b2-4c53-836e-7d55b4b62efd`::Granted request
>Thread-15944::DEBUG::2013-03-18 08:03:42,053::task::794::TaskManager.Task::(resourceAcquired) Task=`ce2863e0-8a72-45ac-880b-73b5dc9225c9`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-15944::DEBUG::2013-03-18 08:03:42,053::task::957::TaskManager.Task::(_decref) Task=`ce2863e0-8a72-45ac-880b-73b5dc9225c9`::ref 1 aborting False
>Thread-15944::INFO::2013-03-18 08:03:42,056::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15944::DEBUG::2013-03-18 08:03:42,056::task::1151::TaskManager.Task::(prepare) Task=`ce2863e0-8a72-45ac-880b-73b5dc9225c9`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15944::DEBUG::2013-03-18 08:03:42,057::task::568::TaskManager.Task::(_updateState) Task=`ce2863e0-8a72-45ac-880b-73b5dc9225c9`::moving from state preparing -> state finished
>Thread-15944::DEBUG::2013-03-18 08:03:42,057::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-15944::DEBUG::2013-03-18 08:03:42,057::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15944::DEBUG::2013-03-18 08:03:42,058::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15944::DEBUG::2013-03-18 08:03:42,058::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15944::DEBUG::2013-03-18 08:03:42,059::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15944::DEBUG::2013-03-18 08:03:42,059::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15944::DEBUG::2013-03-18 08:03:42,059::task::957::TaskManager.Task::(_decref) Task=`ce2863e0-8a72-45ac-880b-73b5dc9225c9`::ref 0 aborting False
>Thread-15945::DEBUG::2013-03-18 08:03:43,097::task::568::TaskManager.Task::(_updateState) Task=`800ada55-e52b-46b7-b90b-992634a2723d`::moving from state init -> state preparing
>Thread-15945::INFO::2013-03-18 08:03:43,104::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15945::INFO::2013-03-18 08:03:43,105::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0488228797913', 'lastCheck': '1.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0128548145294', 'lastCheck': '4.3', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0690608024597', 'lastCheck': '7.2', 'code': 0, 'valid': True}}
>Thread-15945::DEBUG::2013-03-18 08:03:43,105::task::1151::TaskManager.Task::(prepare) Task=`800ada55-e52b-46b7-b90b-992634a2723d`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0488228797913', 'lastCheck': '1.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0128548145294', 'lastCheck': '4.3', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0690608024597', 'lastCheck': '7.2', 'code': 0, 'valid': True}}
>Thread-15945::DEBUG::2013-03-18 08:03:43,105::task::568::TaskManager.Task::(_updateState) Task=`800ada55-e52b-46b7-b90b-992634a2723d`::moving from state preparing -> state finished
>Thread-15945::DEBUG::2013-03-18 08:03:43,106::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15945::DEBUG::2013-03-18 08:03:43,106::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15945::DEBUG::2013-03-18 08:03:43,106::task::957::TaskManager.Task::(_decref) Task=`800ada55-e52b-46b7-b90b-992634a2723d`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:03:43,155::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:03:43,346::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0852968 s, 12.0 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:03:45,367::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:03:45,566::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911339 s, 11.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:03:45,942::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain f33e50cc-eb0a-4720-857d-bc7001a72567
>Thread-22::DEBUG::2013-03-18 08:03:45,945::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-22::DEBUG::2013-03-18 08:03:45,946::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:03:45,983::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00906522 s, 226 kB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:03:45,984::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00906522 s, 226 kB/s'], size: 2048
>Thread-22::DEBUG::2013-03-18 08:03:45,985::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-22::WARNING::2013-03-18 08:03:45,985::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-22::DEBUG::2013-03-18 08:03:46,000::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-22::DEBUG::2013-03-18 08:03:46,001::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 08:03:46,002::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 08:03:46,002::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 08:03:46,003::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 08:03:46,003::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-22::DEBUG::2013-03-18 08:03:46,004::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:03:46,530::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:03:46,534::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-22::DEBUG::2013-03-18 08:03:46,535::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['SDUUID=f33e50cc-eb0a-4720-857d-bc7001a72567', 'VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKRENEWALINTERVALSEC=5', 'LOCKPOLICY=', 'PHYBLKSIZE=512', 'VGUUID=1xnHct-ntxm-y8d3-aw7o-fWAK-nVed-iOxPM0', 'ROLE=Regular', 'DESCRIPTION=vms', 'IOOPTIMEOUTSEC=1', 'CLASS=Data', u'PV0=pv:36001405e2fe1e87706442878688ebb17,uuid:8lSHbP-x8OK-pvrr-apvZ-hmfd-n4f8-Tgc67T,pestart:0,pecount:397,mapoffset:0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'MASTER_VERSION=0', '_SHA_CKSUM=b0169a9bcf7b0b907edcce296ae2d72f8bd79e99']
>Thread-22::WARNING::2013-03-18 08:03:46,538::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_imageNS already registered
>Thread-22::WARNING::2013-03-18 08:03:46,540::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_volumeNS already registered
>Thread-22::WARNING::2013-03-18 08:03:46,540::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace f33e50cc-eb0a-4720-857d-bc7001a72567_lvmActivationNS already registered
>Thread-22::DEBUG::2013-03-18 08:03:46,542::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " f33e50cc-eb0a-4720-857d-bc7001a72567' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:03:47,004::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:03:47,007::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:03:47,041::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00597575 s, 685 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:03:47,589::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:03:47,792::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0977484 s, 10.5 MB/s\n'; <rc> = 0
>Thread-21::DEBUG::2013-03-18 08:03:48,782::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 08:03:48,784::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-21::DEBUG::2013-03-18 08:03:48,785::misc::83::Storage.Misc.excCmd::(<lambda>) u'/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 848fa27a-00f3-4bb4-b4da-101cd912b66e' (cwd None)
>Thread-21::DEBUG::2013-03-18 08:03:49,298::misc::83::Storage.Misc.excCmd::(<lambda>) FAILED: <err> = '  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found\n'; <rc> = 5
>Thread-21::WARNING::2013-03-18 08:03:49,305::lvm::378::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] ['  Volume group "848fa27a-00f3-4bb4-b4da-101cd912b66e" not found']
>Thread-21::DEBUG::2013-03-18 08:03:49,306::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-21::DEBUG::2013-03-18 08:03:49,354::fileSD::131::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.1.147:_rhev_isos/848fa27a-00f3-4bb4-b4da-101cd912b66e
>Thread-21::DEBUG::2013-03-18 08:03:49,355::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend
>Thread-21::DEBUG::2013-03-18 08:03:49,364::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Iso', 'DESCRIPTION=iso', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'REMOTE_PATH=10.34.1.147:/rhev/isos', 'ROLE=Regular', 'SDUUID=848fa27a-00f3-4bb4-b4da-101cd912b66e', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=82f0495b643c327dba06677f3f072dc015b7597c']
>Thread-21::DEBUG::2013-03-18 08:03:49,368::fileSD::503::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images []
>Thread-21::WARNING::2013-03-18 08:03:49,368::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 848fa27a-00f3-4bb4-b4da-101cd912b66e_imageNS already registered
>Thread-21::WARNING::2013-03-18 08:03:49,369::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 848fa27a-00f3-4bb4-b4da-101cd912b66e_volumeNS already registered
>Dummy-15786::DEBUG::2013-03-18 08:03:49,813::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:03:49,995::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0856026 s, 12.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:03:51,891::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:03:51,947::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0163459 s, 251 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:03:52,016::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Thread-15952::DEBUG::2013-03-18 08:03:52,197::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15952::DEBUG::2013-03-18 08:03:52,206::task::568::TaskManager.Task::(_updateState) Task=`0a248ea5-a3c8-4a4a-8bfe-9b5c5d21ecc5`::moving from state init -> state preparing
>Thread-15952::INFO::2013-03-18 08:03:52,210::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15952::INFO::2013-03-18 08:03:52,213::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15952::DEBUG::2013-03-18 08:03:52,214::task::1151::TaskManager.Task::(prepare) Task=`0a248ea5-a3c8-4a4a-8bfe-9b5c5d21ecc5`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15952::DEBUG::2013-03-18 08:03:52,215::task::568::TaskManager.Task::(_updateState) Task=`0a248ea5-a3c8-4a4a-8bfe-9b5c5d21ecc5`::moving from state preparing -> state finished
>Thread-15952::DEBUG::2013-03-18 08:03:52,215::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15952::DEBUG::2013-03-18 08:03:52,216::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15952::DEBUG::2013-03-18 08:03:52,218::task::957::TaskManager.Task::(_decref) Task=`0a248ea5-a3c8-4a4a-8bfe-9b5c5d21ecc5`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:03:52,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0824905 s, 12.4 MB/s\n'; <rc> = 0
>Thread-15953::DEBUG::2013-03-18 08:03:52,244::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15953::DEBUG::2013-03-18 08:03:52,247::task::568::TaskManager.Task::(_updateState) Task=`4bdf73c4-3313-4bbd-b3db-43c8adf06ab3`::moving from state init -> state preparing
>Thread-15953::INFO::2013-03-18 08:03:52,248::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15953::DEBUG::2013-03-18 08:03:52,251::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`b97197fe-abd1-48e1-8155-2c9954bbdd79`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15953::DEBUG::2013-03-18 08:03:52,252::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-15953::DEBUG::2013-03-18 08:03:52,254::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15953::DEBUG::2013-03-18 08:03:52,254::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`b97197fe-abd1-48e1-8155-2c9954bbdd79`::Granted request
>Thread-15953::DEBUG::2013-03-18 08:03:52,260::task::794::TaskManager.Task::(resourceAcquired) Task=`4bdf73c4-3313-4bbd-b3db-43c8adf06ab3`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-15953::DEBUG::2013-03-18 08:03:52,260::task::957::TaskManager.Task::(_decref) Task=`4bdf73c4-3313-4bbd-b3db-43c8adf06ab3`::ref 1 aborting False
>Thread-15953::INFO::2013-03-18 08:03:52,275::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15953::DEBUG::2013-03-18 08:03:52,278::task::1151::TaskManager.Task::(prepare) Task=`4bdf73c4-3313-4bbd-b3db-43c8adf06ab3`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15953::DEBUG::2013-03-18 08:03:52,281::task::568::TaskManager.Task::(_updateState) Task=`4bdf73c4-3313-4bbd-b3db-43c8adf06ab3`::moving from state preparing -> state finished
>Thread-15953::DEBUG::2013-03-18 08:03:52,282::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-15953::DEBUG::2013-03-18 08:03:52,282::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15953::DEBUG::2013-03-18 08:03:52,283::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15953::DEBUG::2013-03-18 08:03:52,285::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15953::DEBUG::2013-03-18 08:03:52,285::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15953::DEBUG::2013-03-18 08:03:52,286::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15953::DEBUG::2013-03-18 08:03:52,287::task::957::TaskManager.Task::(_decref) Task=`4bdf73c4-3313-4bbd-b3db-43c8adf06ab3`::ref 0 aborting False
>Thread-15954::DEBUG::2013-03-18 08:03:53,349::task::568::TaskManager.Task::(_updateState) Task=`084b6bc3-ceab-4db6-bed6-924acf9cfab6`::moving from state init -> state preparing
>Thread-15954::INFO::2013-03-18 08:03:53,349::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15954::INFO::2013-03-18 08:03:53,350::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0579948425293', 'lastCheck': '1.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00755310058594', 'lastCheck': '3.9', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0364799499512', 'lastCheck': '6.3', 'code': 0, 'valid': True}}
>Thread-15954::DEBUG::2013-03-18 08:03:53,350::task::1151::TaskManager.Task::(prepare) Task=`084b6bc3-ceab-4db6-bed6-924acf9cfab6`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0579948425293', 'lastCheck': '1.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00755310058594', 'lastCheck': '3.9', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0364799499512', 'lastCheck': '6.3', 'code': 0, 'valid': True}}
>Thread-15954::DEBUG::2013-03-18 08:03:53,350::task::568::TaskManager.Task::(_updateState) Task=`084b6bc3-ceab-4db6-bed6-924acf9cfab6`::moving from state preparing -> state finished
>Thread-15954::DEBUG::2013-03-18 08:03:53,351::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15954::DEBUG::2013-03-18 08:03:53,351::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15954::DEBUG::2013-03-18 08:03:53,351::task::957::TaskManager.Task::(_decref) Task=`084b6bc3-ceab-4db6-bed6-924acf9cfab6`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:03:54,294::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:03:54,467::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0718877 s, 14.2 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:03:56,487::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:03:56,663::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0786506 s, 13.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:03:57,051::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:03:57,078::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000779981 s, 5.3 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:03:58,684::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:03:58,858::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0756005 s, 13.5 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:04:00,878::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:04:01,067::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0890505 s, 11.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:04:01,996::domainMonitor::170::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 4bf968ef-020a-47ee-a1ae-f85eaeb8751a
>Thread-20::DEBUG::2013-03-18 08:04:01,998::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with LvMetadataRW backend
>Thread-20::DEBUG::2013-03-18 08:04:01,999::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct skip=0 bs=2048 if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:04:02,033::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n2048 bytes (2.0 kB) copied, 0.00704981 s, 291 kB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:04:02,035::misc::316::Storage.Misc::(validateDDBytes) err: ['1+0 records in', '1+0 records out', '2048 bytes (2.0 kB) copied, 0.00704981 s, 291 kB/s'], size: 2048
>Thread-20::DEBUG::2013-03-18 08:04:02,035::persistentDict::234::Storage.PersistentDict::(refresh) read lines (LvMetadataRW)=[]
>Thread-20::WARNING::2013-03-18 08:04:02,035::persistentDict::256::Storage.PersistentDict::(refresh) data has no embedded checksum - trust it as it is
>Thread-20::DEBUG::2013-03-18 08:04:02,037::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with VGTagMetadataRW backend
>Thread-20::DEBUG::2013-03-18 08:04:02,038::lvm::488::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 08:04:02,039::lvm::490::OperationMutex::(_invalidatevgs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 08:04:02,039::lvm::498::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 08:04:02,040::lvm::510::OperationMutex::(_invalidatelvs) Operation 'lvm invalidate operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 08:04:02,040::lvm::373::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex
>Thread-20::DEBUG::2013-03-18 08:04:02,041::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgs --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " --noheadings --units b --nosuffix --separator | -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-15960::DEBUG::2013-03-18 08:04:02,425::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15960::DEBUG::2013-03-18 08:04:02,430::task::568::TaskManager.Task::(_updateState) Task=`6ba5446d-8b68-4580-a49f-37c7812c1e3c`::moving from state init -> state preparing
>Thread-15960::INFO::2013-03-18 08:04:02,431::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-20::DEBUG::2013-03-18 08:04:02,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:04:02,530::lvm::402::OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex
>Thread-20::DEBUG::2013-03-18 08:04:02,531::persistentDict::234::Storage.PersistentDict::(refresh) read lines (VGTagMetadataRW)=['VERSION=3', 'TYPE=ISCSI', 'LOGBLKSIZE=512', u'PV0=pv:3600140558f280d707b243749b07609a3,uuid:APghh2-N3f4-j6xQ-hM22-piv4-jufW-8W4sJR,pestart:0,pecount:77,mapoffset:0', 'LEASERETRIES=3', 'DESCRIPTION=msd', 'LOCKRENEWALINTERVALSEC=5', 'PHYBLKSIZE=512', 'SDUUID=4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'CLASS=Data', 'VGUUID=AJo5kZ-6FF3-2lZR-JREU-9sce-505P-otPxC3', 'LOCKPOLICY=', 'LEASETIMESEC=60', 'IOOPTIMEOUTSEC=10', 'MASTER_VERSION=1', 'ROLE=Master', 'POOL_UUID=a1112cc0-b4af-4396-a3b6-2a3916293a6b', 'POOL_DESCRIPTION=iscis', u'POOL_DOMAINS=4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'POOL_SPM_ID=2', '_SHA_CKSUM=9b6c31302edce2645b9361d66bf5bc8091db2bcb', 'POOL_SPM_LVER=1']
>Thread-20::WARNING::2013-03-18 08:04:02,533::sd::361::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_imageNS already registered
>Thread-20::WARNING::2013-03-18 08:04:02,534::sd::369::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_volumeNS already registered
>Thread-20::WARNING::2013-03-18 08:04:02,535::blockSD::425::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 4bf968ef-020a-47ee-a1ae-f85eaeb8751a_lvmActivationNS already registered
>Thread-15960::INFO::2013-03-18 08:04:02,538::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15960::DEBUG::2013-03-18 08:04:02,540::task::1151::TaskManager.Task::(prepare) Task=`6ba5446d-8b68-4580-a49f-37c7812c1e3c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-20::DEBUG::2013-03-18 08:04:02,541::misc::83::Storage.Misc.excCmd::(<lambda>) '/usr/bin/sudo -n /sbin/lvm vgck --config " devices { preferred_names = [\\"^/dev/mapper/\\"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 filter = [ \'a%1ATA_QEMU_HARDDISK_QM00001|36001405398701597f844f4eb099342d0|3600140558f280d707b243749b07609a3|36001405e2fe1e87706442878688ebb17%\', \'r%.*%\' ] }  global {  locking_type=1  prioritise_write_locks=1  wait_for_locks=1 }  backup {  retain_min = 50  retain_days = 0 } " 4bf968ef-020a-47ee-a1ae-f85eaeb8751a' (cwd None)
>Thread-15960::DEBUG::2013-03-18 08:04:02,541::task::568::TaskManager.Task::(_updateState) Task=`6ba5446d-8b68-4580-a49f-37c7812c1e3c`::moving from state preparing -> state finished
>Thread-15960::DEBUG::2013-03-18 08:04:02,566::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15960::DEBUG::2013-03-18 08:04:02,567::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15960::DEBUG::2013-03-18 08:04:02,569::task::957::TaskManager.Task::(_decref) Task=`6ba5446d-8b68-4580-a49f-37c7812c1e3c`::ref 0 aborting False
>Thread-15961::DEBUG::2013-03-18 08:04:02,591::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15961::DEBUG::2013-03-18 08:04:02,593::task::568::TaskManager.Task::(_updateState) Task=`6d171a6a-2b5c-4772-a27b-e65528c47220`::moving from state init -> state preparing
>Thread-15961::INFO::2013-03-18 08:04:02,594::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15961::DEBUG::2013-03-18 08:04:02,596::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`46edef7b-68e1-4e17-92a8-2712bdf181a8`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15961::DEBUG::2013-03-18 08:04:02,596::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-15961::DEBUG::2013-03-18 08:04:02,597::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15961::DEBUG::2013-03-18 08:04:02,597::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`46edef7b-68e1-4e17-92a8-2712bdf181a8`::Granted request
>Thread-15961::DEBUG::2013-03-18 08:04:02,598::task::794::TaskManager.Task::(resourceAcquired) Task=`6d171a6a-2b5c-4772-a27b-e65528c47220`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-15961::DEBUG::2013-03-18 08:04:02,599::task::957::TaskManager.Task::(_decref) Task=`6d171a6a-2b5c-4772-a27b-e65528c47220`::ref 1 aborting False
>Thread-15961::INFO::2013-03-18 08:04:02,603::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15961::DEBUG::2013-03-18 08:04:02,603::task::1151::TaskManager.Task::(prepare) Task=`6d171a6a-2b5c-4772-a27b-e65528c47220`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15961::DEBUG::2013-03-18 08:04:02,603::task::568::TaskManager.Task::(_updateState) Task=`6d171a6a-2b5c-4772-a27b-e65528c47220`::moving from state preparing -> state finished
>Thread-15961::DEBUG::2013-03-18 08:04:02,604::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-15961::DEBUG::2013-03-18 08:04:02,604::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15961::DEBUG::2013-03-18 08:04:02,604::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15961::DEBUG::2013-03-18 08:04:02,605::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15961::DEBUG::2013-03-18 08:04:02,605::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15961::DEBUG::2013-03-18 08:04:02,605::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15961::DEBUG::2013-03-18 08:04:02,606::task::957::TaskManager.Task::(_decref) Task=`6d171a6a-2b5c-4772-a27b-e65528c47220`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 08:04:03,016::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = ''; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:04:03,017::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:04:03,059::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0146001 s, 281 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:04:03,089::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:04:03,290::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.084378 s, 12.1 MB/s\n'; <rc> = 0
>Thread-15962::DEBUG::2013-03-18 08:04:03,580::task::568::TaskManager.Task::(_updateState) Task=`7d6e5371-352b-44ac-9d99-cc3f2ab403b2`::moving from state init -> state preparing
>Thread-15962::INFO::2013-03-18 08:04:03,581::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15962::INFO::2013-03-18 08:04:03,582::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0444710254669', 'lastCheck': '0.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00792002677917', 'lastCheck': '4.1', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.029079914093', 'lastCheck': '6.5', 'code': 0, 'valid': True}}
>Thread-15962::DEBUG::2013-03-18 08:04:03,583::task::1151::TaskManager.Task::(prepare) Task=`7d6e5371-352b-44ac-9d99-cc3f2ab403b2`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0444710254669', 'lastCheck': '0.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00792002677917', 'lastCheck': '4.1', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.029079914093', 'lastCheck': '6.5', 'code': 0, 'valid': True}}
>Thread-15962::DEBUG::2013-03-18 08:04:03,583::task::568::TaskManager.Task::(_updateState) Task=`7d6e5371-352b-44ac-9d99-cc3f2ab403b2`::moving from state preparing -> state finished
>Thread-15962::DEBUG::2013-03-18 08:04:03,583::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15962::DEBUG::2013-03-18 08:04:03,583::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15962::DEBUG::2013-03-18 08:04:03,584::task::957::TaskManager.Task::(_decref) Task=`7d6e5371-352b-44ac-9d99-cc3f2ab403b2`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:04:05,310::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:04:05,476::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0692491 s, 14.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:04:07,085::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:04:07,116::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000928014 s, 4.4 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:04:07,499::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:04:07,678::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0806551 s, 12.7 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:04:09,703::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:04:09,932::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.117117 s, 8.7 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:04:11,953::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:04:12,136::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0858367 s, 11.9 MB/s\n'; <rc> = 0
>Thread-15968::DEBUG::2013-03-18 08:04:12,720::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15968::DEBUG::2013-03-18 08:04:12,722::task::568::TaskManager.Task::(_updateState) Task=`4e836605-b805-49d4-aa11-48e4778a8d55`::moving from state init -> state preparing
>Thread-15968::INFO::2013-03-18 08:04:12,723::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15968::INFO::2013-03-18 08:04:12,724::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15968::DEBUG::2013-03-18 08:04:12,725::task::1151::TaskManager.Task::(prepare) Task=`4e836605-b805-49d4-aa11-48e4778a8d55`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15968::DEBUG::2013-03-18 08:04:12,725::task::568::TaskManager.Task::(_updateState) Task=`4e836605-b805-49d4-aa11-48e4778a8d55`::moving from state preparing -> state finished
>Thread-15968::DEBUG::2013-03-18 08:04:12,725::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15968::DEBUG::2013-03-18 08:04:12,726::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15968::DEBUG::2013-03-18 08:04:12,726::task::957::TaskManager.Task::(_decref) Task=`4e836605-b805-49d4-aa11-48e4778a8d55`::ref 0 aborting False
>Thread-15969::DEBUG::2013-03-18 08:04:12,749::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15969::DEBUG::2013-03-18 08:04:12,750::task::568::TaskManager.Task::(_updateState) Task=`7e91000a-5cb4-4da1-bcdb-5b30b3141dbd`::moving from state init -> state preparing
>Thread-15969::INFO::2013-03-18 08:04:12,750::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15969::DEBUG::2013-03-18 08:04:12,752::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`f9514fd5-fb7a-4604-93ea-f1c07608898b`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15969::DEBUG::2013-03-18 08:04:12,752::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-15969::DEBUG::2013-03-18 08:04:12,753::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15969::DEBUG::2013-03-18 08:04:12,753::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`f9514fd5-fb7a-4604-93ea-f1c07608898b`::Granted request
>Thread-15969::DEBUG::2013-03-18 08:04:12,754::task::794::TaskManager.Task::(resourceAcquired) Task=`7e91000a-5cb4-4da1-bcdb-5b30b3141dbd`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-15969::DEBUG::2013-03-18 08:04:12,754::task::957::TaskManager.Task::(_decref) Task=`7e91000a-5cb4-4da1-bcdb-5b30b3141dbd`::ref 1 aborting False
>Thread-15969::INFO::2013-03-18 08:04:12,758::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15969::DEBUG::2013-03-18 08:04:12,758::task::1151::TaskManager.Task::(prepare) Task=`7e91000a-5cb4-4da1-bcdb-5b30b3141dbd`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15969::DEBUG::2013-03-18 08:04:12,758::task::568::TaskManager.Task::(_updateState) Task=`7e91000a-5cb4-4da1-bcdb-5b30b3141dbd`::moving from state preparing -> state finished
>Thread-15969::DEBUG::2013-03-18 08:04:12,759::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-15969::DEBUG::2013-03-18 08:04:12,759::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15969::DEBUG::2013-03-18 08:04:12,760::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15969::DEBUG::2013-03-18 08:04:12,760::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15969::DEBUG::2013-03-18 08:04:12,760::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15969::DEBUG::2013-03-18 08:04:12,761::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15969::DEBUG::2013-03-18 08:04:12,761::task::957::TaskManager.Task::(_decref) Task=`7e91000a-5cb4-4da1-bcdb-5b30b3141dbd`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 08:04:13,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:04:13,211::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0167212 s, 245 kB/s\n'; <rc> = 0
>Thread-15970::DEBUG::2013-03-18 08:04:13,867::task::568::TaskManager.Task::(_updateState) Task=`8b374843-8a31-437a-bb65-6e991d028fd2`::moving from state init -> state preparing
>Thread-15970::INFO::2013-03-18 08:04:13,868::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15970::INFO::2013-03-18 08:04:13,869::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0611138343811', 'lastCheck': '0.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00915503501892', 'lastCheck': '4.3', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0327928066254', 'lastCheck': '6.7', 'code': 0, 'valid': True}}
>Thread-15970::DEBUG::2013-03-18 08:04:13,869::task::1151::TaskManager.Task::(prepare) Task=`8b374843-8a31-437a-bb65-6e991d028fd2`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0611138343811', 'lastCheck': '0.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00915503501892', 'lastCheck': '4.3', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0327928066254', 'lastCheck': '6.7', 'code': 0, 'valid': True}}
>Thread-15970::DEBUG::2013-03-18 08:04:13,869::task::568::TaskManager.Task::(_updateState) Task=`8b374843-8a31-437a-bb65-6e991d028fd2`::moving from state preparing -> state finished
>Thread-15970::DEBUG::2013-03-18 08:04:13,869::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15970::DEBUG::2013-03-18 08:04:13,870::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15970::DEBUG::2013-03-18 08:04:13,870::task::957::TaskManager.Task::(_decref) Task=`8b374843-8a31-437a-bb65-6e991d028fd2`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:04:14,157::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:04:14,337::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0803928 s, 12.7 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:04:16,363::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:04:16,536::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0784299 s, 13.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:04:17,123::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:04:17,164::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124387 s, 329 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:04:18,557::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:04:18,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.107002 s, 9.6 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:04:20,784::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:04:20,975::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0888175 s, 11.5 MB/s\n'; <rc> = 0
>Thread-15976::DEBUG::2013-03-18 08:04:22,874::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15976::DEBUG::2013-03-18 08:04:22,876::task::568::TaskManager.Task::(_updateState) Task=`a4e6bdbf-7e61-4d67-a573-38defa9261f1`::moving from state init -> state preparing
>Thread-15976::INFO::2013-03-18 08:04:22,877::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15976::INFO::2013-03-18 08:04:22,878::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15976::DEBUG::2013-03-18 08:04:22,878::task::1151::TaskManager.Task::(prepare) Task=`a4e6bdbf-7e61-4d67-a573-38defa9261f1`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15976::DEBUG::2013-03-18 08:04:22,878::task::568::TaskManager.Task::(_updateState) Task=`a4e6bdbf-7e61-4d67-a573-38defa9261f1`::moving from state preparing -> state finished
>Thread-15976::DEBUG::2013-03-18 08:04:22,879::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15976::DEBUG::2013-03-18 08:04:22,879::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15976::DEBUG::2013-03-18 08:04:22,879::task::957::TaskManager.Task::(_decref) Task=`a4e6bdbf-7e61-4d67-a573-38defa9261f1`::ref 0 aborting False
>Thread-15977::DEBUG::2013-03-18 08:04:22,901::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15977::DEBUG::2013-03-18 08:04:22,902::task::568::TaskManager.Task::(_updateState) Task=`fda3bfbf-65af-4e5b-8328-d6b23ef099d5`::moving from state init -> state preparing
>Thread-15977::INFO::2013-03-18 08:04:22,903::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15977::DEBUG::2013-03-18 08:04:22,904::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`e58283be-4547-47c1-a59c-d7b6b4cec222`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15977::DEBUG::2013-03-18 08:04:22,904::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-15977::DEBUG::2013-03-18 08:04:22,905::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15977::DEBUG::2013-03-18 08:04:22,905::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`e58283be-4547-47c1-a59c-d7b6b4cec222`::Granted request
>Thread-15977::DEBUG::2013-03-18 08:04:22,906::task::794::TaskManager.Task::(resourceAcquired) Task=`fda3bfbf-65af-4e5b-8328-d6b23ef099d5`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-15977::DEBUG::2013-03-18 08:04:22,906::task::957::TaskManager.Task::(_decref) Task=`fda3bfbf-65af-4e5b-8328-d6b23ef099d5`::ref 1 aborting False
>Thread-15977::INFO::2013-03-18 08:04:22,909::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15977::DEBUG::2013-03-18 08:04:22,909::task::1151::TaskManager.Task::(prepare) Task=`fda3bfbf-65af-4e5b-8328-d6b23ef099d5`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15977::DEBUG::2013-03-18 08:04:22,910::task::568::TaskManager.Task::(_updateState) Task=`fda3bfbf-65af-4e5b-8328-d6b23ef099d5`::moving from state preparing -> state finished
>Thread-15977::DEBUG::2013-03-18 08:04:22,910::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-15977::DEBUG::2013-03-18 08:04:22,910::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15977::DEBUG::2013-03-18 08:04:22,911::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15977::DEBUG::2013-03-18 08:04:22,911::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15977::DEBUG::2013-03-18 08:04:22,911::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15977::DEBUG::2013-03-18 08:04:22,911::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15977::DEBUG::2013-03-18 08:04:22,912::task::957::TaskManager.Task::(_decref) Task=`fda3bfbf-65af-4e5b-8328-d6b23ef099d5`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:04:22,995::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:04:23,178::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.083193 s, 12.3 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:04:23,251::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:04:23,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00678826 s, 603 kB/s\n'; <rc> = 0
>Thread-15978::DEBUG::2013-03-18 08:04:24,144::task::568::TaskManager.Task::(_updateState) Task=`eafc7081-d4bf-416f-a8cb-59bc86967d8a`::moving from state init -> state preparing
>Thread-15978::INFO::2013-03-18 08:04:24,145::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15978::INFO::2013-03-18 08:04:24,146::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0510470867157', 'lastCheck': '0.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00845885276794', 'lastCheck': '4.6', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0430059432983', 'lastCheck': '7.0', 'code': 0, 'valid': True}}
>Thread-15978::DEBUG::2013-03-18 08:04:24,146::task::1151::TaskManager.Task::(prepare) Task=`eafc7081-d4bf-416f-a8cb-59bc86967d8a`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0510470867157', 'lastCheck': '0.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00845885276794', 'lastCheck': '4.6', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0430059432983', 'lastCheck': '7.0', 'code': 0, 'valid': True}}
>Thread-15978::DEBUG::2013-03-18 08:04:24,146::task::568::TaskManager.Task::(_updateState) Task=`eafc7081-d4bf-416f-a8cb-59bc86967d8a`::moving from state preparing -> state finished
>Thread-15978::DEBUG::2013-03-18 08:04:24,147::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15978::DEBUG::2013-03-18 08:04:24,147::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15978::DEBUG::2013-03-18 08:04:24,147::task::957::TaskManager.Task::(_decref) Task=`eafc7081-d4bf-416f-a8cb-59bc86967d8a`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:04:25,200::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:04:25,377::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0783262 s, 13.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:04:27,172::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:04:27,200::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000790662 s, 5.2 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:04:27,398::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:04:27,592::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0947219 s, 10.8 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:04:29,613::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:04:29,807::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0854991 s, 12.0 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:04:31,829::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:04:32,005::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0754649 s, 13.6 MB/s\n'; <rc> = 0
>Thread-15984::DEBUG::2013-03-18 08:04:33,033::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15984::DEBUG::2013-03-18 08:04:33,035::task::568::TaskManager.Task::(_updateState) Task=`0119912e-5ce7-4522-a574-3a2e23234583`::moving from state init -> state preparing
>Thread-15984::INFO::2013-03-18 08:04:33,036::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15984::INFO::2013-03-18 08:04:33,038::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15984::DEBUG::2013-03-18 08:04:33,038::task::1151::TaskManager.Task::(prepare) Task=`0119912e-5ce7-4522-a574-3a2e23234583`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15984::DEBUG::2013-03-18 08:04:33,038::task::568::TaskManager.Task::(_updateState) Task=`0119912e-5ce7-4522-a574-3a2e23234583`::moving from state preparing -> state finished
>Thread-15984::DEBUG::2013-03-18 08:04:33,039::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15984::DEBUG::2013-03-18 08:04:33,039::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15984::DEBUG::2013-03-18 08:04:33,039::task::957::TaskManager.Task::(_decref) Task=`0119912e-5ce7-4522-a574-3a2e23234583`::ref 0 aborting False
>Thread-15985::DEBUG::2013-03-18 08:04:33,062::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15985::DEBUG::2013-03-18 08:04:33,063::task::568::TaskManager.Task::(_updateState) Task=`0f617b25-f4cf-421c-8bae-8e124e671c87`::moving from state init -> state preparing
>Thread-15985::INFO::2013-03-18 08:04:33,063::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15985::DEBUG::2013-03-18 08:04:33,065::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`6dbeeab4-f75e-405c-a99a-0b63ac1258d6`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15985::DEBUG::2013-03-18 08:04:33,065::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-15985::DEBUG::2013-03-18 08:04:33,066::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15985::DEBUG::2013-03-18 08:04:33,066::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`6dbeeab4-f75e-405c-a99a-0b63ac1258d6`::Granted request
>Thread-15985::DEBUG::2013-03-18 08:04:33,067::task::794::TaskManager.Task::(resourceAcquired) Task=`0f617b25-f4cf-421c-8bae-8e124e671c87`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-15985::DEBUG::2013-03-18 08:04:33,068::task::957::TaskManager.Task::(_decref) Task=`0f617b25-f4cf-421c-8bae-8e124e671c87`::ref 1 aborting False
>Thread-15985::INFO::2013-03-18 08:04:33,071::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15985::DEBUG::2013-03-18 08:04:33,071::task::1151::TaskManager.Task::(prepare) Task=`0f617b25-f4cf-421c-8bae-8e124e671c87`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15985::DEBUG::2013-03-18 08:04:33,072::task::568::TaskManager.Task::(_updateState) Task=`0f617b25-f4cf-421c-8bae-8e124e671c87`::moving from state preparing -> state finished
>Thread-15985::DEBUG::2013-03-18 08:04:33,072::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-15985::DEBUG::2013-03-18 08:04:33,073::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15985::DEBUG::2013-03-18 08:04:33,073::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15985::DEBUG::2013-03-18 08:04:33,073::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15985::DEBUG::2013-03-18 08:04:33,074::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15985::DEBUG::2013-03-18 08:04:33,074::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15985::DEBUG::2013-03-18 08:04:33,074::task::957::TaskManager.Task::(_decref) Task=`0f617b25-f4cf-421c-8bae-8e124e671c87`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 08:04:33,346::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:04:33,394::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0205969 s, 199 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:04:34,026::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:04:34,217::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0944545 s, 10.8 MB/s\n'; <rc> = 0
>Thread-15986::DEBUG::2013-03-18 08:04:34,389::task::568::TaskManager.Task::(_updateState) Task=`936c92d7-964d-41e5-814b-bf053f65889a`::moving from state init -> state preparing
>Thread-15986::INFO::2013-03-18 08:04:34,390::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15986::INFO::2013-03-18 08:04:34,391::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0500929355621', 'lastCheck': '1.0', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0121150016785', 'lastCheck': '4.7', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0302331447601', 'lastCheck': '7.2', 'code': 0, 'valid': True}}
>Thread-15986::DEBUG::2013-03-18 08:04:34,391::task::1151::TaskManager.Task::(prepare) Task=`936c92d7-964d-41e5-814b-bf053f65889a`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0500929355621', 'lastCheck': '1.0', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0121150016785', 'lastCheck': '4.7', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0302331447601', 'lastCheck': '7.2', 'code': 0, 'valid': True}}
>Thread-15986::DEBUG::2013-03-18 08:04:34,391::task::568::TaskManager.Task::(_updateState) Task=`936c92d7-964d-41e5-814b-bf053f65889a`::moving from state preparing -> state finished
>Thread-15986::DEBUG::2013-03-18 08:04:34,392::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15986::DEBUG::2013-03-18 08:04:34,392::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15986::DEBUG::2013-03-18 08:04:34,392::task::957::TaskManager.Task::(_decref) Task=`936c92d7-964d-41e5-814b-bf053f65889a`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:04:36,239::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:04:36,462::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0805505 s, 12.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:04:37,207::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:04:37,235::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000703713 s, 5.8 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:04:38,483::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:04:38,661::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0826646 s, 12.4 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:04:40,682::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:04:40,864::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0834624 s, 12.3 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:04:42,884::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:04:43,046::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0655913 s, 15.6 MB/s\n'; <rc> = 0
>Thread-15992::DEBUG::2013-03-18 08:04:43,195::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15992::DEBUG::2013-03-18 08:04:43,197::task::568::TaskManager.Task::(_updateState) Task=`d304cebd-e676-40d1-894d-be46978fff33`::moving from state init -> state preparing
>Thread-15992::INFO::2013-03-18 08:04:43,198::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15992::INFO::2013-03-18 08:04:43,199::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15992::DEBUG::2013-03-18 08:04:43,199::task::1151::TaskManager.Task::(prepare) Task=`d304cebd-e676-40d1-894d-be46978fff33`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-15992::DEBUG::2013-03-18 08:04:43,199::task::568::TaskManager.Task::(_updateState) Task=`d304cebd-e676-40d1-894d-be46978fff33`::moving from state preparing -> state finished
>Thread-15992::DEBUG::2013-03-18 08:04:43,200::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15992::DEBUG::2013-03-18 08:04:43,200::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15992::DEBUG::2013-03-18 08:04:43,200::task::957::TaskManager.Task::(_decref) Task=`d304cebd-e676-40d1-894d-be46978fff33`::ref 0 aborting False
>Thread-15993::DEBUG::2013-03-18 08:04:43,225::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-15993::DEBUG::2013-03-18 08:04:43,226::task::568::TaskManager.Task::(_updateState) Task=`977e8bd1-226d-4b06-8520-0b15ae17966f`::moving from state init -> state preparing
>Thread-15993::INFO::2013-03-18 08:04:43,227::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-15993::DEBUG::2013-03-18 08:04:43,228::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`ef907199-726a-4368-a093-9f81f2510822`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-15993::DEBUG::2013-03-18 08:04:43,228::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-15993::DEBUG::2013-03-18 08:04:43,229::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-15993::DEBUG::2013-03-18 08:04:43,230::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`ef907199-726a-4368-a093-9f81f2510822`::Granted request
>Thread-15993::DEBUG::2013-03-18 08:04:43,231::task::794::TaskManager.Task::(resourceAcquired) Task=`977e8bd1-226d-4b06-8520-0b15ae17966f`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-15993::DEBUG::2013-03-18 08:04:43,231::task::957::TaskManager.Task::(_decref) Task=`977e8bd1-226d-4b06-8520-0b15ae17966f`::ref 1 aborting False
>Thread-15993::INFO::2013-03-18 08:04:43,234::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15993::DEBUG::2013-03-18 08:04:43,234::task::1151::TaskManager.Task::(prepare) Task=`977e8bd1-226d-4b06-8520-0b15ae17966f`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-15993::DEBUG::2013-03-18 08:04:43,235::task::568::TaskManager.Task::(_updateState) Task=`977e8bd1-226d-4b06-8520-0b15ae17966f`::moving from state preparing -> state finished
>Thread-15993::DEBUG::2013-03-18 08:04:43,235::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-15993::DEBUG::2013-03-18 08:04:43,235::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15993::DEBUG::2013-03-18 08:04:43,235::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-15993::DEBUG::2013-03-18 08:04:43,236::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-15993::DEBUG::2013-03-18 08:04:43,236::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-15993::DEBUG::2013-03-18 08:04:43,236::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-15993::DEBUG::2013-03-18 08:04:43,237::task::957::TaskManager.Task::(_decref) Task=`977e8bd1-226d-4b06-8520-0b15ae17966f`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 08:04:43,441::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:04:43,488::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00864377 s, 474 kB/s\n'; <rc> = 0
>Thread-15994::DEBUG::2013-03-18 08:04:44,696::task::568::TaskManager.Task::(_updateState) Task=`5b355748-9a10-40b6-8c86-fb4a045db442`::moving from state init -> state preparing
>Thread-15994::INFO::2013-03-18 08:04:44,697::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-15994::INFO::2013-03-18 08:04:44,698::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0489161014557', 'lastCheck': '1.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0101499557495', 'lastCheck': '5.0', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.030513048172', 'lastCheck': '7.5', 'code': 0, 'valid': True}}
>Thread-15994::DEBUG::2013-03-18 08:04:44,698::task::1151::TaskManager.Task::(prepare) Task=`5b355748-9a10-40b6-8c86-fb4a045db442`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0489161014557', 'lastCheck': '1.2', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0101499557495', 'lastCheck': '5.0', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.030513048172', 'lastCheck': '7.5', 'code': 0, 'valid': True}}
>Thread-15994::DEBUG::2013-03-18 08:04:44,698::task::568::TaskManager.Task::(_updateState) Task=`5b355748-9a10-40b6-8c86-fb4a045db442`::moving from state preparing -> state finished
>Thread-15994::DEBUG::2013-03-18 08:04:44,699::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-15994::DEBUG::2013-03-18 08:04:44,699::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-15994::DEBUG::2013-03-18 08:04:44,699::task::957::TaskManager.Task::(_decref) Task=`5b355748-9a10-40b6-8c86-fb4a045db442`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:04:45,068::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:04:45,248::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0832167 s, 12.3 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:04:47,241::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:04:47,278::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:04:47,327::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000782122 s, 5.2 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:04:47,498::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.113217 s, 9.0 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:04:49,518::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:04:49,704::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0882601 s, 11.6 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:04:51,726::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:04:51,931::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0911366 s, 11.2 MB/s\n'; <rc> = 0
>Thread-16000::DEBUG::2013-03-18 08:04:53,348::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16000::DEBUG::2013-03-18 08:04:53,350::task::568::TaskManager.Task::(_updateState) Task=`70de1426-9fb1-4320-afdc-2931af0f3c46`::moving from state init -> state preparing
>Thread-16000::INFO::2013-03-18 08:04:53,351::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16000::INFO::2013-03-18 08:04:53,352::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16000::DEBUG::2013-03-18 08:04:53,352::task::1151::TaskManager.Task::(prepare) Task=`70de1426-9fb1-4320-afdc-2931af0f3c46`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16000::DEBUG::2013-03-18 08:04:53,353::task::568::TaskManager.Task::(_updateState) Task=`70de1426-9fb1-4320-afdc-2931af0f3c46`::moving from state preparing -> state finished
>Thread-16000::DEBUG::2013-03-18 08:04:53,353::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16000::DEBUG::2013-03-18 08:04:53,353::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16000::DEBUG::2013-03-18 08:04:53,354::task::957::TaskManager.Task::(_decref) Task=`70de1426-9fb1-4320-afdc-2931af0f3c46`::ref 0 aborting False
>Thread-16001::DEBUG::2013-03-18 08:04:53,376::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16001::DEBUG::2013-03-18 08:04:53,377::task::568::TaskManager.Task::(_updateState) Task=`f29be289-7e8d-4069-8dba-e56dff9e1886`::moving from state init -> state preparing
>Thread-16001::INFO::2013-03-18 08:04:53,377::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16001::DEBUG::2013-03-18 08:04:53,378::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`05a45c2a-1ca6-4364-a075-9fdbece5d11e`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-16001::DEBUG::2013-03-18 08:04:53,379::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-16001::DEBUG::2013-03-18 08:04:53,379::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-16001::DEBUG::2013-03-18 08:04:53,380::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`05a45c2a-1ca6-4364-a075-9fdbece5d11e`::Granted request
>Thread-16001::DEBUG::2013-03-18 08:04:53,380::task::794::TaskManager.Task::(resourceAcquired) Task=`f29be289-7e8d-4069-8dba-e56dff9e1886`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-16001::DEBUG::2013-03-18 08:04:53,381::task::957::TaskManager.Task::(_decref) Task=`f29be289-7e8d-4069-8dba-e56dff9e1886`::ref 1 aborting False
>Thread-16001::INFO::2013-03-18 08:04:53,383::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16001::DEBUG::2013-03-18 08:04:53,384::task::1151::TaskManager.Task::(prepare) Task=`f29be289-7e8d-4069-8dba-e56dff9e1886`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16001::DEBUG::2013-03-18 08:04:53,384::task::568::TaskManager.Task::(_updateState) Task=`f29be289-7e8d-4069-8dba-e56dff9e1886`::moving from state preparing -> state finished
>Thread-16001::DEBUG::2013-03-18 08:04:53,384::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-16001::DEBUG::2013-03-18 08:04:53,385::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16001::DEBUG::2013-03-18 08:04:53,385::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-16001::DEBUG::2013-03-18 08:04:53,385::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-16001::DEBUG::2013-03-18 08:04:53,386::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-16001::DEBUG::2013-03-18 08:04:53,386::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-16001::DEBUG::2013-03-18 08:04:53,386::task::957::TaskManager.Task::(_decref) Task=`f29be289-7e8d-4069-8dba-e56dff9e1886`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 08:04:53,535::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:04:53,591::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0156994 s, 261 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:04:53,953::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:04:54,142::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0898584 s, 11.4 MB/s\n'; <rc> = 0
>Thread-16002::DEBUG::2013-03-18 08:04:54,950::task::568::TaskManager.Task::(_updateState) Task=`7d665bba-ff97-44f1-85c7-ca4b7ebe7096`::moving from state init -> state preparing
>Thread-16002::INFO::2013-03-18 08:04:54,951::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-16002::INFO::2013-03-18 08:04:54,952::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0576031208038', 'lastCheck': '1.3', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0089590549469', 'lastCheck': '5.2', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0877909660339', 'lastCheck': '7.6', 'code': 0, 'valid': True}}
>Thread-16002::DEBUG::2013-03-18 08:04:54,952::task::1151::TaskManager.Task::(prepare) Task=`7d665bba-ff97-44f1-85c7-ca4b7ebe7096`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0576031208038', 'lastCheck': '1.3', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0089590549469', 'lastCheck': '5.2', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0877909660339', 'lastCheck': '7.6', 'code': 0, 'valid': True}}
>Thread-16002::DEBUG::2013-03-18 08:04:54,953::task::568::TaskManager.Task::(_updateState) Task=`7d665bba-ff97-44f1-85c7-ca4b7ebe7096`::moving from state preparing -> state finished
>Thread-16002::DEBUG::2013-03-18 08:04:54,953::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16002::DEBUG::2013-03-18 08:04:54,953::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16002::DEBUG::2013-03-18 08:04:54,953::task::957::TaskManager.Task::(_decref) Task=`7d665bba-ff97-44f1-85c7-ca4b7ebe7096`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:04:56,163::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:04:56,361::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0798459 s, 12.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:04:57,334::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:04:57,373::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00394552 s, 1.0 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:04:58,383::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:04:58,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.105511 s, 9.7 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:05:00,603::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:05:00,794::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0905666 s, 11.3 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:05:02,814::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:05:02,994::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0802019 s, 12.8 MB/s\n'; <rc> = 0
>Thread-16008::DEBUG::2013-03-18 08:05:03,506::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16008::DEBUG::2013-03-18 08:05:03,509::task::568::TaskManager.Task::(_updateState) Task=`cd792ff4-a654-49d3-abe6-cf5a8f40e667`::moving from state init -> state preparing
>Thread-16008::INFO::2013-03-18 08:05:03,510::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16008::INFO::2013-03-18 08:05:03,511::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16008::DEBUG::2013-03-18 08:05:03,511::task::1151::TaskManager.Task::(prepare) Task=`cd792ff4-a654-49d3-abe6-cf5a8f40e667`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16008::DEBUG::2013-03-18 08:05:03,512::task::568::TaskManager.Task::(_updateState) Task=`cd792ff4-a654-49d3-abe6-cf5a8f40e667`::moving from state preparing -> state finished
>Thread-16008::DEBUG::2013-03-18 08:05:03,512::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16008::DEBUG::2013-03-18 08:05:03,513::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16008::DEBUG::2013-03-18 08:05:03,513::task::957::TaskManager.Task::(_decref) Task=`cd792ff4-a654-49d3-abe6-cf5a8f40e667`::ref 0 aborting False
>Thread-16009::DEBUG::2013-03-18 08:05:03,528::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16009::DEBUG::2013-03-18 08:05:03,529::task::568::TaskManager.Task::(_updateState) Task=`4c148c2b-3d85-4e6c-b6d4-5662ef690276`::moving from state init -> state preparing
>Thread-16009::INFO::2013-03-18 08:05:03,529::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16009::DEBUG::2013-03-18 08:05:03,530::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`8a29f615-399b-4796-bfe7-d06e825f63b5`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-16009::DEBUG::2013-03-18 08:05:03,530::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-16009::DEBUG::2013-03-18 08:05:03,531::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-16009::DEBUG::2013-03-18 08:05:03,531::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`8a29f615-399b-4796-bfe7-d06e825f63b5`::Granted request
>Thread-16009::DEBUG::2013-03-18 08:05:03,532::task::794::TaskManager.Task::(resourceAcquired) Task=`4c148c2b-3d85-4e6c-b6d4-5662ef690276`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-16009::DEBUG::2013-03-18 08:05:03,532::task::957::TaskManager.Task::(_decref) Task=`4c148c2b-3d85-4e6c-b6d4-5662ef690276`::ref 1 aborting False
>Thread-16009::INFO::2013-03-18 08:05:03,535::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16009::DEBUG::2013-03-18 08:05:03,535::task::1151::TaskManager.Task::(prepare) Task=`4c148c2b-3d85-4e6c-b6d4-5662ef690276`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16009::DEBUG::2013-03-18 08:05:03,536::task::568::TaskManager.Task::(_updateState) Task=`4c148c2b-3d85-4e6c-b6d4-5662ef690276`::moving from state preparing -> state finished
>Thread-16009::DEBUG::2013-03-18 08:05:03,536::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-16009::DEBUG::2013-03-18 08:05:03,536::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16009::DEBUG::2013-03-18 08:05:03,536::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-16009::DEBUG::2013-03-18 08:05:03,537::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-16009::DEBUG::2013-03-18 08:05:03,537::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-16009::DEBUG::2013-03-18 08:05:03,537::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-16009::DEBUG::2013-03-18 08:05:03,538::task::957::TaskManager.Task::(_decref) Task=`4c148c2b-3d85-4e6c-b6d4-5662ef690276`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 08:05:03,637::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:05:03,693::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0201119 s, 204 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:05:05,016::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:05:05,212::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0688517 s, 14.9 MB/s\n'; <rc> = 0
>Thread-16010::DEBUG::2013-03-18 08:05:05,246::task::568::TaskManager.Task::(_updateState) Task=`94278cea-58a5-48ad-9dc8-40905d8b3e12`::moving from state init -> state preparing
>Thread-16010::INFO::2013-03-18 08:05:05,247::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-16010::INFO::2013-03-18 08:05:05,248::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0584709644318', 'lastCheck': '1.5', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00858092308044', 'lastCheck': '5.4', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0404651165009', 'lastCheck': '7.9', 'code': 0, 'valid': True}}
>Thread-16010::DEBUG::2013-03-18 08:05:05,248::task::1151::TaskManager.Task::(prepare) Task=`94278cea-58a5-48ad-9dc8-40905d8b3e12`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0584709644318', 'lastCheck': '1.5', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00858092308044', 'lastCheck': '5.4', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0404651165009', 'lastCheck': '7.9', 'code': 0, 'valid': True}}
>Thread-16010::DEBUG::2013-03-18 08:05:05,249::task::568::TaskManager.Task::(_updateState) Task=`94278cea-58a5-48ad-9dc8-40905d8b3e12`::moving from state preparing -> state finished
>Thread-16010::DEBUG::2013-03-18 08:05:05,249::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16010::DEBUG::2013-03-18 08:05:05,249::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16010::DEBUG::2013-03-18 08:05:05,249::task::957::TaskManager.Task::(_decref) Task=`94278cea-58a5-48ad-9dc8-40905d8b3e12`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:05:07,267::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:05:07,573::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:05:07,918::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00678033 s, 604 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:05:08,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.602531 s, 1.7 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:05:10,039::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:05:10,224::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0880277 s, 11.6 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:05:12,245::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:05:12,419::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0761288 s, 13.5 MB/s\n'; <rc> = 0
>Thread-16015::DEBUG::2013-03-18 08:05:13,658::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16015::DEBUG::2013-03-18 08:05:13,660::task::568::TaskManager.Task::(_updateState) Task=`11107a11-79e5-4847-b917-e6bef7c629c2`::moving from state init -> state preparing
>Thread-16015::INFO::2013-03-18 08:05:13,661::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16015::INFO::2013-03-18 08:05:13,662::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16015::DEBUG::2013-03-18 08:05:13,663::task::1151::TaskManager.Task::(prepare) Task=`11107a11-79e5-4847-b917-e6bef7c629c2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16015::DEBUG::2013-03-18 08:05:13,663::task::568::TaskManager.Task::(_updateState) Task=`11107a11-79e5-4847-b917-e6bef7c629c2`::moving from state preparing -> state finished
>Thread-16015::DEBUG::2013-03-18 08:05:13,663::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16015::DEBUG::2013-03-18 08:05:13,663::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16015::DEBUG::2013-03-18 08:05:13,664::task::957::TaskManager.Task::(_decref) Task=`11107a11-79e5-4847-b917-e6bef7c629c2`::ref 0 aborting False
>Thread-16016::DEBUG::2013-03-18 08:05:13,685::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16016::DEBUG::2013-03-18 08:05:13,685::task::568::TaskManager.Task::(_updateState) Task=`984eecdc-3d01-42e4-b16c-3322444e2318`::moving from state init -> state preparing
>Thread-16016::INFO::2013-03-18 08:05:13,686::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16016::DEBUG::2013-03-18 08:05:13,687::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`d8428689-27d6-4682-b513-ad7aa6a3a243`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-16016::DEBUG::2013-03-18 08:05:13,687::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-16016::DEBUG::2013-03-18 08:05:13,687::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-16016::DEBUG::2013-03-18 08:05:13,688::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`d8428689-27d6-4682-b513-ad7aa6a3a243`::Granted request
>Thread-16016::DEBUG::2013-03-18 08:05:13,688::task::794::TaskManager.Task::(resourceAcquired) Task=`984eecdc-3d01-42e4-b16c-3322444e2318`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-16016::DEBUG::2013-03-18 08:05:13,688::task::957::TaskManager.Task::(_decref) Task=`984eecdc-3d01-42e4-b16c-3322444e2318`::ref 1 aborting False
>Thread-16016::INFO::2013-03-18 08:05:13,690::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16016::DEBUG::2013-03-18 08:05:13,691::task::1151::TaskManager.Task::(prepare) Task=`984eecdc-3d01-42e4-b16c-3322444e2318`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16016::DEBUG::2013-03-18 08:05:13,691::task::568::TaskManager.Task::(_updateState) Task=`984eecdc-3d01-42e4-b16c-3322444e2318`::moving from state preparing -> state finished
>Thread-16016::DEBUG::2013-03-18 08:05:13,691::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-16016::DEBUG::2013-03-18 08:05:13,691::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16016::DEBUG::2013-03-18 08:05:13,692::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-16016::DEBUG::2013-03-18 08:05:13,692::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-16016::DEBUG::2013-03-18 08:05:13,692::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-16016::DEBUG::2013-03-18 08:05:13,692::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-16016::DEBUG::2013-03-18 08:05:13,693::task::957::TaskManager.Task::(_decref) Task=`984eecdc-3d01-42e4-b16c-3322444e2318`::ref 0 aborting False
>Thread-20::DEBUG::2013-03-18 08:05:13,743::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:05:13,806::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0329532 s, 124 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:05:14,442::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:05:14,653::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.113853 s, 9.0 MB/s\n'; <rc> = 0
>Thread-16018::DEBUG::2013-03-18 08:05:15,755::task::568::TaskManager.Task::(_updateState) Task=`910bd150-15ec-4af5-8f47-062d62846b11`::moving from state init -> state preparing
>Thread-16018::INFO::2013-03-18 08:05:15,756::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-16018::INFO::2013-03-18 08:05:15,757::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0645098686218', 'lastCheck': '1.9', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00989413261414', 'lastCheck': '5.9', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.37230014801', 'lastCheck': '7.8', 'code': 0, 'valid': True}}
>Thread-16018::DEBUG::2013-03-18 08:05:15,757::task::1151::TaskManager.Task::(prepare) Task=`910bd150-15ec-4af5-8f47-062d62846b11`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0645098686218', 'lastCheck': '1.9', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00989413261414', 'lastCheck': '5.9', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.37230014801', 'lastCheck': '7.8', 'code': 0, 'valid': True}}
>Thread-16018::DEBUG::2013-03-18 08:05:15,757::task::568::TaskManager.Task::(_updateState) Task=`910bd150-15ec-4af5-8f47-062d62846b11`::moving from state preparing -> state finished
>Thread-16018::DEBUG::2013-03-18 08:05:15,758::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16018::DEBUG::2013-03-18 08:05:15,758::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16018::DEBUG::2013-03-18 08:05:15,758::task::957::TaskManager.Task::(_decref) Task=`910bd150-15ec-4af5-8f47-062d62846b11`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:05:16,673::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:05:16,848::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0773022 s, 13.2 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:05:17,954::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:05:17,984::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00086999 s, 4.7 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:05:18,870::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:05:19,056::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0919899 s, 11.1 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:05:21,076::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:05:21,245::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0706833 s, 14.5 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:05:23,266::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:05:23,443::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0783178 s, 13.1 MB/s\n'; <rc> = 0
>Thread-16023::DEBUG::2013-03-18 08:05:23,816::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16023::DEBUG::2013-03-18 08:05:23,818::task::568::TaskManager.Task::(_updateState) Task=`ce720e39-0bf9-46d0-8fa6-cfaba5ec09c4`::moving from state init -> state preparing
>Thread-16023::INFO::2013-03-18 08:05:23,819::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16023::INFO::2013-03-18 08:05:23,820::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16023::DEBUG::2013-03-18 08:05:23,821::task::1151::TaskManager.Task::(prepare) Task=`ce720e39-0bf9-46d0-8fa6-cfaba5ec09c4`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16023::DEBUG::2013-03-18 08:05:23,821::task::568::TaskManager.Task::(_updateState) Task=`ce720e39-0bf9-46d0-8fa6-cfaba5ec09c4`::moving from state preparing -> state finished
>Thread-16023::DEBUG::2013-03-18 08:05:23,821::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16023::DEBUG::2013-03-18 08:05:23,821::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16023::DEBUG::2013-03-18 08:05:23,822::task::957::TaskManager.Task::(_decref) Task=`ce720e39-0bf9-46d0-8fa6-cfaba5ec09c4`::ref 0 aborting False
>Thread-16024::DEBUG::2013-03-18 08:05:23,842::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16024::DEBUG::2013-03-18 08:05:23,843::task::568::TaskManager.Task::(_updateState) Task=`6b020da3-3c17-4008-b57b-0dd903d960f8`::moving from state init -> state preparing
>Thread-16024::INFO::2013-03-18 08:05:23,844::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16024::DEBUG::2013-03-18 08:05:23,845::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`f6ba0787-1ae7-4f9f-b371-377eba96a243`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-16024::DEBUG::2013-03-18 08:05:23,845::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-16024::DEBUG::2013-03-18 08:05:23,846::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-16024::DEBUG::2013-03-18 08:05:23,846::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`f6ba0787-1ae7-4f9f-b371-377eba96a243`::Granted request
>Thread-16024::DEBUG::2013-03-18 08:05:23,850::task::794::TaskManager.Task::(resourceAcquired) Task=`6b020da3-3c17-4008-b57b-0dd903d960f8`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-16024::DEBUG::2013-03-18 08:05:23,850::task::957::TaskManager.Task::(_decref) Task=`6b020da3-3c17-4008-b57b-0dd903d960f8`::ref 1 aborting False
>Thread-20::DEBUG::2013-03-18 08:05:23,861::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-16024::INFO::2013-03-18 08:05:23,915::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-20::DEBUG::2013-03-18 08:05:23,925::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0124938 s, 328 kB/s\n'; <rc> = 0
>Thread-16024::DEBUG::2013-03-18 08:05:23,929::task::1151::TaskManager.Task::(prepare) Task=`6b020da3-3c17-4008-b57b-0dd903d960f8`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16024::DEBUG::2013-03-18 08:05:23,934::task::568::TaskManager.Task::(_updateState) Task=`6b020da3-3c17-4008-b57b-0dd903d960f8`::moving from state preparing -> state finished
>Thread-16024::DEBUG::2013-03-18 08:05:23,939::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-16024::DEBUG::2013-03-18 08:05:23,942::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16024::DEBUG::2013-03-18 08:05:23,950::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-16024::DEBUG::2013-03-18 08:05:23,954::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-16024::DEBUG::2013-03-18 08:05:23,955::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-16024::DEBUG::2013-03-18 08:05:23,957::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-16024::DEBUG::2013-03-18 08:05:23,961::task::957::TaskManager.Task::(_decref) Task=`6b020da3-3c17-4008-b57b-0dd903d960f8`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:05:25,463::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:05:25,635::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.076336 s, 13.4 MB/s\n'; <rc> = 0
>Thread-16026::DEBUG::2013-03-18 08:05:26,083::task::568::TaskManager.Task::(_updateState) Task=`4bba746d-c76e-4370-8efd-fdd69b9ace17`::moving from state init -> state preparing
>Thread-16026::INFO::2013-03-18 08:05:26,084::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-16026::INFO::2013-03-18 08:05:26,085::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0718879699707', 'lastCheck': '2.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0115549564362', 'lastCheck': '6.1', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0320091247559', 'lastCheck': '8.1', 'code': 0, 'valid': True}}
>Thread-16026::DEBUG::2013-03-18 08:05:26,085::task::1151::TaskManager.Task::(prepare) Task=`4bba746d-c76e-4370-8efd-fdd69b9ace17`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0718879699707', 'lastCheck': '2.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0115549564362', 'lastCheck': '6.1', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0320091247559', 'lastCheck': '8.1', 'code': 0, 'valid': True}}
>Thread-16026::DEBUG::2013-03-18 08:05:26,085::task::568::TaskManager.Task::(_updateState) Task=`4bba746d-c76e-4370-8efd-fdd69b9ace17`::moving from state preparing -> state finished
>Thread-16026::DEBUG::2013-03-18 08:05:26,086::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16026::DEBUG::2013-03-18 08:05:26,086::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16026::DEBUG::2013-03-18 08:05:26,086::task::957::TaskManager.Task::(_decref) Task=`4bba746d-c76e-4370-8efd-fdd69b9ace17`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:05:27,657::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:05:27,878::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.117122 s, 8.7 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:05:27,992::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:05:28,020::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000750447 s, 5.5 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:05:29,898::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:05:30,113::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.117024 s, 8.8 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:05:32,134::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:05:32,348::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.088405 s, 11.6 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:05:34,018::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:05:34,046::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000934693 s, 4.4 MB/s\n'; <rc> = 0
>Thread-16031::DEBUG::2013-03-18 08:05:34,111::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16031::DEBUG::2013-03-18 08:05:34,113::task::568::TaskManager.Task::(_updateState) Task=`c6f10a28-44a4-4f47-9565-48e1735a6266`::moving from state init -> state preparing
>Thread-16031::INFO::2013-03-18 08:05:34,113::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16031::INFO::2013-03-18 08:05:34,114::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16031::DEBUG::2013-03-18 08:05:34,114::task::1151::TaskManager.Task::(prepare) Task=`c6f10a28-44a4-4f47-9565-48e1735a6266`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16031::DEBUG::2013-03-18 08:05:34,115::task::568::TaskManager.Task::(_updateState) Task=`c6f10a28-44a4-4f47-9565-48e1735a6266`::moving from state preparing -> state finished
>Thread-16031::DEBUG::2013-03-18 08:05:34,115::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16031::DEBUG::2013-03-18 08:05:34,115::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16031::DEBUG::2013-03-18 08:05:34,115::task::957::TaskManager.Task::(_decref) Task=`c6f10a28-44a4-4f47-9565-48e1735a6266`::ref 0 aborting False
>Thread-16032::DEBUG::2013-03-18 08:05:34,137::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16032::DEBUG::2013-03-18 08:05:34,138::task::568::TaskManager.Task::(_updateState) Task=`9963b319-13ba-44f5-90a3-953e765137b6`::moving from state init -> state preparing
>Thread-16032::INFO::2013-03-18 08:05:34,139::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16032::DEBUG::2013-03-18 08:05:34,140::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`9204eac0-422e-4ed5-b434-fa6689f30dff`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-16032::DEBUG::2013-03-18 08:05:34,140::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-16032::DEBUG::2013-03-18 08:05:34,141::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-16032::DEBUG::2013-03-18 08:05:34,142::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`9204eac0-422e-4ed5-b434-fa6689f30dff`::Granted request
>Thread-16032::DEBUG::2013-03-18 08:05:34,143::task::794::TaskManager.Task::(resourceAcquired) Task=`9963b319-13ba-44f5-90a3-953e765137b6`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-16032::DEBUG::2013-03-18 08:05:34,143::task::957::TaskManager.Task::(_decref) Task=`9963b319-13ba-44f5-90a3-953e765137b6`::ref 1 aborting False
>Thread-16032::INFO::2013-03-18 08:05:34,146::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16032::DEBUG::2013-03-18 08:05:34,146::task::1151::TaskManager.Task::(prepare) Task=`9963b319-13ba-44f5-90a3-953e765137b6`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16032::DEBUG::2013-03-18 08:05:34,147::task::568::TaskManager.Task::(_updateState) Task=`9963b319-13ba-44f5-90a3-953e765137b6`::moving from state preparing -> state finished
>Thread-16032::DEBUG::2013-03-18 08:05:34,147::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-16032::DEBUG::2013-03-18 08:05:34,147::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16032::DEBUG::2013-03-18 08:05:34,148::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-16032::DEBUG::2013-03-18 08:05:34,148::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-16032::DEBUG::2013-03-18 08:05:34,149::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-16032::DEBUG::2013-03-18 08:05:34,149::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-16032::DEBUG::2013-03-18 08:05:34,149::task::957::TaskManager.Task::(_decref) Task=`9963b319-13ba-44f5-90a3-953e765137b6`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:05:34,368::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:05:34,544::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0808879 s, 12.7 MB/s\n'; <rc> = 0
>Thread-16034::DEBUG::2013-03-18 08:05:36,349::task::568::TaskManager.Task::(_updateState) Task=`71baa839-6154-40a7-9cef-f117b298c26b`::moving from state init -> state preparing
>Thread-16034::INFO::2013-03-18 08:05:36,350::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-16034::INFO::2013-03-18 08:05:36,351::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0300529003143', 'lastCheck': '2.3', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0123360157013', 'lastCheck': '6.3', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0297379493713', 'lastCheck': '8.3', 'code': 0, 'valid': True}}
>Thread-16034::DEBUG::2013-03-18 08:05:36,352::task::1151::TaskManager.Task::(prepare) Task=`71baa839-6154-40a7-9cef-f117b298c26b`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0300529003143', 'lastCheck': '2.3', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0123360157013', 'lastCheck': '6.3', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0297379493713', 'lastCheck': '8.3', 'code': 0, 'valid': True}}
>Thread-16034::DEBUG::2013-03-18 08:05:36,352::task::568::TaskManager.Task::(_updateState) Task=`71baa839-6154-40a7-9cef-f117b298c26b`::moving from state preparing -> state finished
>Thread-16034::DEBUG::2013-03-18 08:05:36,352::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16034::DEBUG::2013-03-18 08:05:36,353::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16034::DEBUG::2013-03-18 08:05:36,353::task::957::TaskManager.Task::(_decref) Task=`71baa839-6154-40a7-9cef-f117b298c26b`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:05:36,566::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:05:36,763::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0926512 s, 11.1 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:05:38,026::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:05:38,066::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000902389 s, 4.5 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:05:38,785::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:05:38,979::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0844833 s, 12.1 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:05:40,999::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:05:41,167::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.07294 s, 14.0 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:05:43,189::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:05:43,371::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0901305 s, 11.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:05:44,101::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:05:44,145::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0106693 s, 384 kB/s\n'; <rc> = 0
>Thread-16039::DEBUG::2013-03-18 08:05:44,277::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16039::DEBUG::2013-03-18 08:05:44,280::task::568::TaskManager.Task::(_updateState) Task=`e2a9aba0-9a02-4836-8d4c-2ebf0b79d39c`::moving from state init -> state preparing
>Thread-16039::INFO::2013-03-18 08:05:44,281::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16039::INFO::2013-03-18 08:05:44,282::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16039::DEBUG::2013-03-18 08:05:44,282::task::1151::TaskManager.Task::(prepare) Task=`e2a9aba0-9a02-4836-8d4c-2ebf0b79d39c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16039::DEBUG::2013-03-18 08:05:44,282::task::568::TaskManager.Task::(_updateState) Task=`e2a9aba0-9a02-4836-8d4c-2ebf0b79d39c`::moving from state preparing -> state finished
>Thread-16039::DEBUG::2013-03-18 08:05:44,283::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16039::DEBUG::2013-03-18 08:05:44,283::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16039::DEBUG::2013-03-18 08:05:44,283::task::957::TaskManager.Task::(_decref) Task=`e2a9aba0-9a02-4836-8d4c-2ebf0b79d39c`::ref 0 aborting False
>Thread-16040::DEBUG::2013-03-18 08:05:44,308::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16040::DEBUG::2013-03-18 08:05:44,309::task::568::TaskManager.Task::(_updateState) Task=`efcdd3a3-c944-43ba-ae5a-a228b038f87e`::moving from state init -> state preparing
>Thread-16040::INFO::2013-03-18 08:05:44,309::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16040::DEBUG::2013-03-18 08:05:44,311::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`4ebe5520-875e-4940-8f04-76e63679c1ba`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-16040::DEBUG::2013-03-18 08:05:44,311::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-16040::DEBUG::2013-03-18 08:05:44,312::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-16040::DEBUG::2013-03-18 08:05:44,312::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`4ebe5520-875e-4940-8f04-76e63679c1ba`::Granted request
>Thread-16040::DEBUG::2013-03-18 08:05:44,313::task::794::TaskManager.Task::(resourceAcquired) Task=`efcdd3a3-c944-43ba-ae5a-a228b038f87e`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-16040::DEBUG::2013-03-18 08:05:44,314::task::957::TaskManager.Task::(_decref) Task=`efcdd3a3-c944-43ba-ae5a-a228b038f87e`::ref 1 aborting False
>Thread-16040::INFO::2013-03-18 08:05:44,317::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16040::DEBUG::2013-03-18 08:05:44,317::task::1151::TaskManager.Task::(prepare) Task=`efcdd3a3-c944-43ba-ae5a-a228b038f87e`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16040::DEBUG::2013-03-18 08:05:44,317::task::568::TaskManager.Task::(_updateState) Task=`efcdd3a3-c944-43ba-ae5a-a228b038f87e`::moving from state preparing -> state finished
>Thread-16040::DEBUG::2013-03-18 08:05:44,318::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-16040::DEBUG::2013-03-18 08:05:44,318::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16040::DEBUG::2013-03-18 08:05:44,319::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-16040::DEBUG::2013-03-18 08:05:44,319::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-16040::DEBUG::2013-03-18 08:05:44,319::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-16040::DEBUG::2013-03-18 08:05:44,320::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-16040::DEBUG::2013-03-18 08:05:44,320::task::957::TaskManager.Task::(_decref) Task=`efcdd3a3-c944-43ba-ae5a-a228b038f87e`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:05:45,382::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:05:45,582::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.100099 s, 10.2 MB/s\n'; <rc> = 0
>Thread-16042::DEBUG::2013-03-18 08:05:46,613::task::568::TaskManager.Task::(_updateState) Task=`26d5fff6-fc5a-4eaa-938b-633ade109570`::moving from state init -> state preparing
>Thread-16042::INFO::2013-03-18 08:05:46,614::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-16042::INFO::2013-03-18 08:05:46,615::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0461089611053', 'lastCheck': '2.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00927400588989', 'lastCheck': '6.5', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0415940284729', 'lastCheck': '8.5', 'code': 0, 'valid': True}}
>Thread-16042::DEBUG::2013-03-18 08:05:46,615::task::1151::TaskManager.Task::(prepare) Task=`26d5fff6-fc5a-4eaa-938b-633ade109570`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0461089611053', 'lastCheck': '2.4', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00927400588989', 'lastCheck': '6.5', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0415940284729', 'lastCheck': '8.5', 'code': 0, 'valid': True}}
>Thread-16042::DEBUG::2013-03-18 08:05:46,615::task::568::TaskManager.Task::(_updateState) Task=`26d5fff6-fc5a-4eaa-938b-633ade109570`::moving from state preparing -> state finished
>Thread-16042::DEBUG::2013-03-18 08:05:46,615::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16042::DEBUG::2013-03-18 08:05:46,615::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16042::DEBUG::2013-03-18 08:05:46,616::task::957::TaskManager.Task::(_decref) Task=`26d5fff6-fc5a-4eaa-938b-633ade109570`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:05:47,605::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:05:47,786::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0785244 s, 13.0 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:05:48,074::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:05:48,104::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000980186 s, 4.2 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:05:49,806::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:05:49,988::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0829321 s, 12.3 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:05:52,010::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:05:52,199::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0925571 s, 11.1 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:05:54,193::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:05:54,221::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:05:54,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0118909 s, 344 kB/s\n'; <rc> = 0
>Thread-16047::DEBUG::2013-03-18 08:05:54,484::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16047::DEBUG::2013-03-18 08:05:54,493::task::568::TaskManager.Task::(_updateState) Task=`857d885d-fa31-4c9e-9795-e02cbfa001b2`::moving from state init -> state preparing
>Thread-16047::INFO::2013-03-18 08:05:54,497::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16047::INFO::2013-03-18 08:05:54,501::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16047::DEBUG::2013-03-18 08:05:54,502::task::1151::TaskManager.Task::(prepare) Task=`857d885d-fa31-4c9e-9795-e02cbfa001b2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16047::DEBUG::2013-03-18 08:05:54,504::task::568::TaskManager.Task::(_updateState) Task=`857d885d-fa31-4c9e-9795-e02cbfa001b2`::moving from state preparing -> state finished
>Thread-16047::DEBUG::2013-03-18 08:05:54,505::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16047::DEBUG::2013-03-18 08:05:54,507::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16047::DEBUG::2013-03-18 08:05:54,509::task::957::TaskManager.Task::(_decref) Task=`857d885d-fa31-4c9e-9795-e02cbfa001b2`::ref 0 aborting False
>Thread-16048::DEBUG::2013-03-18 08:05:54,546::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16048::DEBUG::2013-03-18 08:05:54,549::task::568::TaskManager.Task::(_updateState) Task=`86f4d441-2a7d-42ae-944f-0bc11cdedbe6`::moving from state init -> state preparing
>Thread-16048::INFO::2013-03-18 08:05:54,552::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16048::DEBUG::2013-03-18 08:05:54,557::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`a6e3b70a-1061-4157-87fc-90736dd49510`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-16048::DEBUG::2013-03-18 08:05:54,559::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-16048::DEBUG::2013-03-18 08:05:54,563::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-16048::DEBUG::2013-03-18 08:05:54,565::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`a6e3b70a-1061-4157-87fc-90736dd49510`::Granted request
>Thread-16048::DEBUG::2013-03-18 08:05:54,568::task::794::TaskManager.Task::(resourceAcquired) Task=`86f4d441-2a7d-42ae-944f-0bc11cdedbe6`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-16048::DEBUG::2013-03-18 08:05:54,569::task::957::TaskManager.Task::(_decref) Task=`86f4d441-2a7d-42ae-944f-0bc11cdedbe6`::ref 1 aborting False
>Thread-16048::INFO::2013-03-18 08:05:54,580::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16048::DEBUG::2013-03-18 08:05:54,582::task::1151::TaskManager.Task::(prepare) Task=`86f4d441-2a7d-42ae-944f-0bc11cdedbe6`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16048::DEBUG::2013-03-18 08:05:54,584::task::568::TaskManager.Task::(_updateState) Task=`86f4d441-2a7d-42ae-944f-0bc11cdedbe6`::moving from state preparing -> state finished
>Thread-16048::DEBUG::2013-03-18 08:05:54,586::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-16048::DEBUG::2013-03-18 08:05:54,587::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16048::DEBUG::2013-03-18 08:05:54,589::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-16048::DEBUG::2013-03-18 08:05:54,591::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-16048::DEBUG::2013-03-18 08:05:54,592::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-16048::DEBUG::2013-03-18 08:05:54,593::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-16048::DEBUG::2013-03-18 08:05:54,597::task::957::TaskManager.Task::(_decref) Task=`86f4d441-2a7d-42ae-944f-0bc11cdedbe6`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:05:54,630::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.183482 s, 5.6 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:05:56,649::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:05:56,823::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0734222 s, 13.9 MB/s\n'; <rc> = 0
>Thread-16050::DEBUG::2013-03-18 08:05:56,856::task::568::TaskManager.Task::(_updateState) Task=`b7cb4de4-14a4-4176-b79a-fe6831259549`::moving from state init -> state preparing
>Thread-16050::INFO::2013-03-18 08:05:56,857::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-16050::INFO::2013-03-18 08:05:56,858::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0732018947601', 'lastCheck': '2.5', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00883507728577', 'lastCheck': '6.7', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0323159694672', 'lastCheck': '8.7', 'code': 0, 'valid': True}}
>Thread-16050::DEBUG::2013-03-18 08:05:56,858::task::1151::TaskManager.Task::(prepare) Task=`b7cb4de4-14a4-4176-b79a-fe6831259549`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0732018947601', 'lastCheck': '2.5', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00883507728577', 'lastCheck': '6.7', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0323159694672', 'lastCheck': '8.7', 'code': 0, 'valid': True}}
>Thread-16050::DEBUG::2013-03-18 08:05:56,859::task::568::TaskManager.Task::(_updateState) Task=`b7cb4de4-14a4-4176-b79a-fe6831259549`::moving from state preparing -> state finished
>Thread-16050::DEBUG::2013-03-18 08:05:56,859::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16050::DEBUG::2013-03-18 08:05:56,859::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16050::DEBUG::2013-03-18 08:05:56,860::task::957::TaskManager.Task::(_decref) Task=`b7cb4de4-14a4-4176-b79a-fe6831259549`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 08:05:58,111::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:05:58,144::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00107641 s, 3.8 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:05:58,852::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:05:59,063::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.109244 s, 9.4 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:06:01,083::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:06:01,265::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0846487 s, 12.1 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:06:03,287::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:06:03,468::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.082579 s, 12.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:06:04,398::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:06:04,437::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0107288 s, 382 kB/s\n'; <rc> = 0
>Thread-16055::DEBUG::2013-03-18 08:06:04,724::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16055::DEBUG::2013-03-18 08:06:04,727::task::568::TaskManager.Task::(_updateState) Task=`70b0a967-0e18-491a-9765-a52257c7d108`::moving from state init -> state preparing
>Thread-16055::INFO::2013-03-18 08:06:04,728::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16055::INFO::2013-03-18 08:06:04,728::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16055::DEBUG::2013-03-18 08:06:04,729::task::1151::TaskManager.Task::(prepare) Task=`70b0a967-0e18-491a-9765-a52257c7d108`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16055::DEBUG::2013-03-18 08:06:04,729::task::568::TaskManager.Task::(_updateState) Task=`70b0a967-0e18-491a-9765-a52257c7d108`::moving from state preparing -> state finished
>Thread-16055::DEBUG::2013-03-18 08:06:04,729::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16055::DEBUG::2013-03-18 08:06:04,730::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16055::DEBUG::2013-03-18 08:06:04,730::task::957::TaskManager.Task::(_decref) Task=`70b0a967-0e18-491a-9765-a52257c7d108`::ref 0 aborting False
>Thread-16056::DEBUG::2013-03-18 08:06:04,751::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16056::DEBUG::2013-03-18 08:06:04,752::task::568::TaskManager.Task::(_updateState) Task=`6305c3f7-4268-4b6e-84e4-b58d52478bc7`::moving from state init -> state preparing
>Thread-16056::INFO::2013-03-18 08:06:04,753::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16056::DEBUG::2013-03-18 08:06:04,754::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`2e505d39-5f58-4181-88a2-3468e516525d`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-16056::DEBUG::2013-03-18 08:06:04,754::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-16056::DEBUG::2013-03-18 08:06:04,755::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-16056::DEBUG::2013-03-18 08:06:04,755::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`2e505d39-5f58-4181-88a2-3468e516525d`::Granted request
>Thread-16056::DEBUG::2013-03-18 08:06:04,756::task::794::TaskManager.Task::(resourceAcquired) Task=`6305c3f7-4268-4b6e-84e4-b58d52478bc7`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-16056::DEBUG::2013-03-18 08:06:04,757::task::957::TaskManager.Task::(_decref) Task=`6305c3f7-4268-4b6e-84e4-b58d52478bc7`::ref 1 aborting False
>Thread-16056::INFO::2013-03-18 08:06:04,759::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16056::DEBUG::2013-03-18 08:06:04,760::task::1151::TaskManager.Task::(prepare) Task=`6305c3f7-4268-4b6e-84e4-b58d52478bc7`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16056::DEBUG::2013-03-18 08:06:04,760::task::568::TaskManager.Task::(_updateState) Task=`6305c3f7-4268-4b6e-84e4-b58d52478bc7`::moving from state preparing -> state finished
>Thread-16056::DEBUG::2013-03-18 08:06:04,760::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-16056::DEBUG::2013-03-18 08:06:04,761::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16056::DEBUG::2013-03-18 08:06:04,761::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-16056::DEBUG::2013-03-18 08:06:04,762::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-16056::DEBUG::2013-03-18 08:06:04,762::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-16056::DEBUG::2013-03-18 08:06:04,762::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-16056::DEBUG::2013-03-18 08:06:04,763::task::957::TaskManager.Task::(_decref) Task=`6305c3f7-4268-4b6e-84e4-b58d52478bc7`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:06:05,492::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:06:05,679::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0848821 s, 12.1 MB/s\n'; <rc> = 0
>Thread-16058::DEBUG::2013-03-18 08:06:07,099::task::568::TaskManager.Task::(_updateState) Task=`674af90c-4238-434a-9ab4-8e6cb944c6d7`::moving from state init -> state preparing
>Thread-16058::INFO::2013-03-18 08:06:07,100::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-16058::INFO::2013-03-18 08:06:07,104::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.04159116745', 'lastCheck': '2.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0100319385529', 'lastCheck': '6.9', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0354280471802', 'lastCheck': '9.0', 'code': 0, 'valid': True}}
>Thread-16058::DEBUG::2013-03-18 08:06:07,105::task::1151::TaskManager.Task::(prepare) Task=`674af90c-4238-434a-9ab4-8e6cb944c6d7`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.04159116745', 'lastCheck': '2.6', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0100319385529', 'lastCheck': '6.9', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0354280471802', 'lastCheck': '9.0', 'code': 0, 'valid': True}}
>Thread-16058::DEBUG::2013-03-18 08:06:07,105::task::568::TaskManager.Task::(_updateState) Task=`674af90c-4238-434a-9ab4-8e6cb944c6d7`::moving from state preparing -> state finished
>Thread-16058::DEBUG::2013-03-18 08:06:07,106::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16058::DEBUG::2013-03-18 08:06:07,106::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16058::DEBUG::2013-03-18 08:06:07,106::task::957::TaskManager.Task::(_decref) Task=`674af90c-4238-434a-9ab4-8e6cb944c6d7`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:06:07,699::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:06:07,884::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0870386 s, 11.8 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:06:08,152::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:06:08,182::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00079674 s, 5.1 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:06:09,905::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:06:10,099::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0970964 s, 10.5 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:06:12,121::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:06:12,304::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.083781 s, 12.2 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:06:14,326::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:06:14,486::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:06:14,527::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0010075 s, 4.1 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:06:14,579::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.088059 s, 11.6 MB/s\n'; <rc> = 0
>Thread-16063::DEBUG::2013-03-18 08:06:14,942::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16063::DEBUG::2013-03-18 08:06:14,944::task::568::TaskManager.Task::(_updateState) Task=`14751d3a-81a8-4863-9127-a76cad5b96d5`::moving from state init -> state preparing
>Thread-16063::INFO::2013-03-18 08:06:14,945::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16063::INFO::2013-03-18 08:06:14,946::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16063::DEBUG::2013-03-18 08:06:14,947::task::1151::TaskManager.Task::(prepare) Task=`14751d3a-81a8-4863-9127-a76cad5b96d5`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16063::DEBUG::2013-03-18 08:06:14,947::task::568::TaskManager.Task::(_updateState) Task=`14751d3a-81a8-4863-9127-a76cad5b96d5`::moving from state preparing -> state finished
>Thread-16063::DEBUG::2013-03-18 08:06:14,947::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16063::DEBUG::2013-03-18 08:06:14,948::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16063::DEBUG::2013-03-18 08:06:14,948::task::957::TaskManager.Task::(_decref) Task=`14751d3a-81a8-4863-9127-a76cad5b96d5`::ref 0 aborting False
>Thread-16064::DEBUG::2013-03-18 08:06:14,970::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16064::DEBUG::2013-03-18 08:06:14,971::task::568::TaskManager.Task::(_updateState) Task=`73282ca9-1b29-4260-9e50-ae7416385502`::moving from state init -> state preparing
>Thread-16064::INFO::2013-03-18 08:06:14,972::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16064::DEBUG::2013-03-18 08:06:14,973::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`e12fc887-3120-4e01-963d-e387eadda1ef`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-16064::DEBUG::2013-03-18 08:06:14,973::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-16064::DEBUG::2013-03-18 08:06:14,974::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-16064::DEBUG::2013-03-18 08:06:14,975::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`e12fc887-3120-4e01-963d-e387eadda1ef`::Granted request
>Thread-16064::DEBUG::2013-03-18 08:06:14,976::task::794::TaskManager.Task::(resourceAcquired) Task=`73282ca9-1b29-4260-9e50-ae7416385502`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-16064::DEBUG::2013-03-18 08:06:14,976::task::957::TaskManager.Task::(_decref) Task=`73282ca9-1b29-4260-9e50-ae7416385502`::ref 1 aborting False
>Thread-16064::INFO::2013-03-18 08:06:14,979::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16064::DEBUG::2013-03-18 08:06:14,979::task::1151::TaskManager.Task::(prepare) Task=`73282ca9-1b29-4260-9e50-ae7416385502`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16064::DEBUG::2013-03-18 08:06:14,980::task::568::TaskManager.Task::(_updateState) Task=`73282ca9-1b29-4260-9e50-ae7416385502`::moving from state preparing -> state finished
>Thread-16064::DEBUG::2013-03-18 08:06:14,980::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-16064::DEBUG::2013-03-18 08:06:14,980::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16064::DEBUG::2013-03-18 08:06:14,981::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-16064::DEBUG::2013-03-18 08:06:14,981::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-16064::DEBUG::2013-03-18 08:06:14,981::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-16064::DEBUG::2013-03-18 08:06:14,982::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-16064::DEBUG::2013-03-18 08:06:14,982::task::957::TaskManager.Task::(_decref) Task=`73282ca9-1b29-4260-9e50-ae7416385502`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:06:16,613::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:06:16,815::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0765416 s, 13.4 MB/s\n'; <rc> = 0
>Thread-16066::DEBUG::2013-03-18 08:06:17,418::task::568::TaskManager.Task::(_updateState) Task=`537b5955-6848-47ab-ba65-1219a609569b`::moving from state init -> state preparing
>Thread-16066::INFO::2013-03-18 08:06:17,419::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-16066::INFO::2013-03-18 08:06:17,419::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0443370342255', 'lastCheck': '2.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0099151134491', 'lastCheck': '7.1', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.031998872757', 'lastCheck': '9.2', 'code': 0, 'valid': True}}
>Thread-16066::DEBUG::2013-03-18 08:06:17,420::task::1151::TaskManager.Task::(prepare) Task=`537b5955-6848-47ab-ba65-1219a609569b`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0443370342255', 'lastCheck': '2.8', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0099151134491', 'lastCheck': '7.1', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.031998872757', 'lastCheck': '9.2', 'code': 0, 'valid': True}}
>Thread-16066::DEBUG::2013-03-18 08:06:17,420::task::568::TaskManager.Task::(_updateState) Task=`537b5955-6848-47ab-ba65-1219a609569b`::moving from state preparing -> state finished
>Thread-16066::DEBUG::2013-03-18 08:06:17,420::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16066::DEBUG::2013-03-18 08:06:17,421::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16066::DEBUG::2013-03-18 08:06:17,421::task::957::TaskManager.Task::(_decref) Task=`537b5955-6848-47ab-ba65-1219a609569b`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 08:06:18,189::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:06:18,218::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000847794 s, 4.8 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:06:18,837::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:06:19,036::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0954881 s, 10.7 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:06:21,057::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:06:21,229::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0733707 s, 14.0 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:06:23,251::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:06:23,446::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0860635 s, 11.9 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:06:24,616::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:06:24,668::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0188937 s, 217 kB/s\n'; <rc> = 0
>Thread-16071::DEBUG::2013-03-18 08:06:25,117::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16071::DEBUG::2013-03-18 08:06:25,118::task::568::TaskManager.Task::(_updateState) Task=`f4837fbf-b14e-4d77-82e1-693f41530ca6`::moving from state init -> state preparing
>Thread-16071::INFO::2013-03-18 08:06:25,119::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16071::INFO::2013-03-18 08:06:25,120::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16071::DEBUG::2013-03-18 08:06:25,120::task::1151::TaskManager.Task::(prepare) Task=`f4837fbf-b14e-4d77-82e1-693f41530ca6`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16071::DEBUG::2013-03-18 08:06:25,120::task::568::TaskManager.Task::(_updateState) Task=`f4837fbf-b14e-4d77-82e1-693f41530ca6`::moving from state preparing -> state finished
>Thread-16071::DEBUG::2013-03-18 08:06:25,121::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16071::DEBUG::2013-03-18 08:06:25,121::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16071::DEBUG::2013-03-18 08:06:25,121::task::957::TaskManager.Task::(_decref) Task=`f4837fbf-b14e-4d77-82e1-693f41530ca6`::ref 0 aborting False
>Thread-16072::DEBUG::2013-03-18 08:06:25,143::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16072::DEBUG::2013-03-18 08:06:25,144::task::568::TaskManager.Task::(_updateState) Task=`400d2dae-ca77-49a9-ada2-fc4a99df9525`::moving from state init -> state preparing
>Thread-16072::INFO::2013-03-18 08:06:25,144::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16072::DEBUG::2013-03-18 08:06:25,146::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`80ceee1e-cdcf-46a8-acdc-cd3aaebaec46`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-16072::DEBUG::2013-03-18 08:06:25,146::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-16072::DEBUG::2013-03-18 08:06:25,147::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-16072::DEBUG::2013-03-18 08:06:25,147::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`80ceee1e-cdcf-46a8-acdc-cd3aaebaec46`::Granted request
>Thread-16072::DEBUG::2013-03-18 08:06:25,148::task::794::TaskManager.Task::(resourceAcquired) Task=`400d2dae-ca77-49a9-ada2-fc4a99df9525`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-16072::DEBUG::2013-03-18 08:06:25,148::task::957::TaskManager.Task::(_decref) Task=`400d2dae-ca77-49a9-ada2-fc4a99df9525`::ref 1 aborting False
>Thread-16072::INFO::2013-03-18 08:06:25,151::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16072::DEBUG::2013-03-18 08:06:25,152::task::1151::TaskManager.Task::(prepare) Task=`400d2dae-ca77-49a9-ada2-fc4a99df9525`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16072::DEBUG::2013-03-18 08:06:25,152::task::568::TaskManager.Task::(_updateState) Task=`400d2dae-ca77-49a9-ada2-fc4a99df9525`::moving from state preparing -> state finished
>Thread-16072::DEBUG::2013-03-18 08:06:25,152::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-16072::DEBUG::2013-03-18 08:06:25,153::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16072::DEBUG::2013-03-18 08:06:25,153::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-16072::DEBUG::2013-03-18 08:06:25,153::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-16072::DEBUG::2013-03-18 08:06:25,154::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-16072::DEBUG::2013-03-18 08:06:25,154::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-16072::DEBUG::2013-03-18 08:06:25,154::task::957::TaskManager.Task::(_decref) Task=`400d2dae-ca77-49a9-ada2-fc4a99df9525`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:06:25,466::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:06:25,640::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0756263 s, 13.5 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:06:27,662::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Thread-16074::DEBUG::2013-03-18 08:06:27,775::task::568::TaskManager.Task::(_updateState) Task=`3b24a27e-56c0-4ce1-9e69-07c2cf1e8379`::moving from state init -> state preparing
>Thread-16074::INFO::2013-03-18 08:06:27,785::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-16074::INFO::2013-03-18 08:06:27,791::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0546569824219', 'lastCheck': '3.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00756597518921', 'lastCheck': '7.5', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0309269428253', 'lastCheck': '9.6', 'code': 0, 'valid': True}}
>Thread-16074::DEBUG::2013-03-18 08:06:27,793::task::1151::TaskManager.Task::(prepare) Task=`3b24a27e-56c0-4ce1-9e69-07c2cf1e8379`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0546569824219', 'lastCheck': '3.1', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00756597518921', 'lastCheck': '7.5', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0309269428253', 'lastCheck': '9.6', 'code': 0, 'valid': True}}
>Thread-16074::DEBUG::2013-03-18 08:06:27,794::task::568::TaskManager.Task::(_updateState) Task=`3b24a27e-56c0-4ce1-9e69-07c2cf1e8379`::moving from state preparing -> state finished
>Thread-16074::DEBUG::2013-03-18 08:06:27,796::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16074::DEBUG::2013-03-18 08:06:27,798::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16074::DEBUG::2013-03-18 08:06:27,799::task::957::TaskManager.Task::(_decref) Task=`3b24a27e-56c0-4ce1-9e69-07c2cf1e8379`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:06:27,940::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.181999 s, 5.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:06:28,223::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:06:28,264::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00187712 s, 2.2 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:06:29,960::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:06:30,112::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.053234 s, 19.2 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:06:32,133::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:06:32,300::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0598417 s, 17.1 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:06:34,321::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:06:34,485::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0664448 s, 15.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:06:34,718::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:06:34,745::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.000764146 s, 5.4 MB/s\n'; <rc> = 0
>Thread-16079::DEBUG::2013-03-18 08:06:35,284::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16079::DEBUG::2013-03-18 08:06:35,286::task::568::TaskManager.Task::(_updateState) Task=`481a073a-42e5-4eaf-8ec0-2ae07f460392`::moving from state init -> state preparing
>Thread-16079::INFO::2013-03-18 08:06:35,287::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16079::INFO::2013-03-18 08:06:35,288::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16079::DEBUG::2013-03-18 08:06:35,289::task::1151::TaskManager.Task::(prepare) Task=`481a073a-42e5-4eaf-8ec0-2ae07f460392`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16079::DEBUG::2013-03-18 08:06:35,289::task::568::TaskManager.Task::(_updateState) Task=`481a073a-42e5-4eaf-8ec0-2ae07f460392`::moving from state preparing -> state finished
>Thread-16079::DEBUG::2013-03-18 08:06:35,289::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16079::DEBUG::2013-03-18 08:06:35,290::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16079::DEBUG::2013-03-18 08:06:35,290::task::957::TaskManager.Task::(_decref) Task=`481a073a-42e5-4eaf-8ec0-2ae07f460392`::ref 0 aborting False
>Thread-16080::DEBUG::2013-03-18 08:06:35,313::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16080::DEBUG::2013-03-18 08:06:35,314::task::568::TaskManager.Task::(_updateState) Task=`e11b5356-75a2-499f-b9bb-970fb4a827cb`::moving from state init -> state preparing
>Thread-16080::INFO::2013-03-18 08:06:35,315::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16080::DEBUG::2013-03-18 08:06:35,316::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`a638135f-9dfd-4cb3-8778-4d74dd5c2166`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-16080::DEBUG::2013-03-18 08:06:35,316::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-16080::DEBUG::2013-03-18 08:06:35,317::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-16080::DEBUG::2013-03-18 08:06:35,317::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`a638135f-9dfd-4cb3-8778-4d74dd5c2166`::Granted request
>Thread-16080::DEBUG::2013-03-18 08:06:35,318::task::794::TaskManager.Task::(resourceAcquired) Task=`e11b5356-75a2-499f-b9bb-970fb4a827cb`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-16080::DEBUG::2013-03-18 08:06:35,319::task::957::TaskManager.Task::(_decref) Task=`e11b5356-75a2-499f-b9bb-970fb4a827cb`::ref 1 aborting False
>Thread-16080::INFO::2013-03-18 08:06:35,321::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16080::DEBUG::2013-03-18 08:06:35,322::task::1151::TaskManager.Task::(prepare) Task=`e11b5356-75a2-499f-b9bb-970fb4a827cb`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16080::DEBUG::2013-03-18 08:06:35,322::task::568::TaskManager.Task::(_updateState) Task=`e11b5356-75a2-499f-b9bb-970fb4a827cb`::moving from state preparing -> state finished
>Thread-16080::DEBUG::2013-03-18 08:06:35,323::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-16080::DEBUG::2013-03-18 08:06:35,323::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16080::DEBUG::2013-03-18 08:06:35,323::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-16080::DEBUG::2013-03-18 08:06:35,324::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-16080::DEBUG::2013-03-18 08:06:35,324::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-16080::DEBUG::2013-03-18 08:06:35,325::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-16080::DEBUG::2013-03-18 08:06:35,325::task::957::TaskManager.Task::(_decref) Task=`e11b5356-75a2-499f-b9bb-970fb4a827cb`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:06:36,505::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:06:36,699::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0903948 s, 11.3 MB/s\n'; <rc> = 0
>Thread-16082::DEBUG::2013-03-18 08:06:38,089::task::568::TaskManager.Task::(_updateState) Task=`f42ee5aa-d43a-4c90-9a12-f4752d8ffe93`::moving from state init -> state preparing
>Thread-16082::INFO::2013-03-18 08:06:38,090::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-16082::INFO::2013-03-18 08:06:38,091::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0288619995117', 'lastCheck': '3.3', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0121438503265', 'lastCheck': '7.7', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0436198711395', 'lastCheck': '9.8', 'code': 0, 'valid': True}}
>Thread-16082::DEBUG::2013-03-18 08:06:38,092::task::1151::TaskManager.Task::(prepare) Task=`f42ee5aa-d43a-4c90-9a12-f4752d8ffe93`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0288619995117', 'lastCheck': '3.3', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0121438503265', 'lastCheck': '7.7', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0436198711395', 'lastCheck': '9.8', 'code': 0, 'valid': True}}
>Thread-16082::DEBUG::2013-03-18 08:06:38,092::task::568::TaskManager.Task::(_updateState) Task=`f42ee5aa-d43a-4c90-9a12-f4752d8ffe93`::moving from state preparing -> state finished
>Thread-16082::DEBUG::2013-03-18 08:06:38,092::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16082::DEBUG::2013-03-18 08:06:38,093::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16082::DEBUG::2013-03-18 08:06:38,093::task::957::TaskManager.Task::(_decref) Task=`f42ee5aa-d43a-4c90-9a12-f4752d8ffe93`::ref 0 aborting False
>Thread-22::DEBUG::2013-03-18 08:06:38,271::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:06:38,316::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00131002 s, 3.1 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:06:38,723::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:06:38,916::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.088336 s, 11.6 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:06:40,936::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:06:41,108::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0699351 s, 14.6 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:06:43,129::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:06:43,336::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.102046 s, 10.0 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:06:44,792::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:06:44,834::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.012524 s, 327 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:06:45,358::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Thread-16087::DEBUG::2013-03-18 08:06:45,466::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16087::DEBUG::2013-03-18 08:06:45,481::task::568::TaskManager.Task::(_updateState) Task=`0d4022e7-5e1d-4e15-bd94-8f32dd80981e`::moving from state init -> state preparing
>Thread-16087::INFO::2013-03-18 08:06:45,487::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16087::INFO::2013-03-18 08:06:45,493::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16087::DEBUG::2013-03-18 08:06:45,494::task::1151::TaskManager.Task::(prepare) Task=`0d4022e7-5e1d-4e15-bd94-8f32dd80981e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16087::DEBUG::2013-03-18 08:06:45,497::task::568::TaskManager.Task::(_updateState) Task=`0d4022e7-5e1d-4e15-bd94-8f32dd80981e`::moving from state preparing -> state finished
>Thread-16087::DEBUG::2013-03-18 08:06:45,498::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16087::DEBUG::2013-03-18 08:06:45,500::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16087::DEBUG::2013-03-18 08:06:45,502::task::957::TaskManager.Task::(_decref) Task=`0d4022e7-5e1d-4e15-bd94-8f32dd80981e`::ref 0 aborting False
>Thread-16088::DEBUG::2013-03-18 08:06:45,536::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16088::DEBUG::2013-03-18 08:06:45,539::task::568::TaskManager.Task::(_updateState) Task=`df0736e5-f020-4af8-a3ee-9e235128b363`::moving from state init -> state preparing
>Thread-16088::INFO::2013-03-18 08:06:45,541::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16088::DEBUG::2013-03-18 08:06:45,548::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`7e55fbb6-8d3c-45cf-8602-20cf4c6e5928`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-16088::DEBUG::2013-03-18 08:06:45,549::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-16088::DEBUG::2013-03-18 08:06:45,553::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-16088::DEBUG::2013-03-18 08:06:45,555::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`7e55fbb6-8d3c-45cf-8602-20cf4c6e5928`::Granted request
>Thread-16088::DEBUG::2013-03-18 08:06:45,559::task::794::TaskManager.Task::(resourceAcquired) Task=`df0736e5-f020-4af8-a3ee-9e235128b363`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-16088::DEBUG::2013-03-18 08:06:45,561::task::957::TaskManager.Task::(_decref) Task=`df0736e5-f020-4af8-a3ee-9e235128b363`::ref 1 aborting False
>Thread-16088::INFO::2013-03-18 08:06:45,575::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16088::DEBUG::2013-03-18 08:06:45,577::task::1151::TaskManager.Task::(prepare) Task=`df0736e5-f020-4af8-a3ee-9e235128b363`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16088::DEBUG::2013-03-18 08:06:45,578::task::568::TaskManager.Task::(_updateState) Task=`df0736e5-f020-4af8-a3ee-9e235128b363`::moving from state preparing -> state finished
>Thread-16088::DEBUG::2013-03-18 08:06:45,579::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-16088::DEBUG::2013-03-18 08:06:45,579::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16088::DEBUG::2013-03-18 08:06:45,586::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-16088::DEBUG::2013-03-18 08:06:45,588::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-16088::DEBUG::2013-03-18 08:06:45,588::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-16088::DEBUG::2013-03-18 08:06:45,588::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-16088::DEBUG::2013-03-18 08:06:45,590::task::957::TaskManager.Task::(_decref) Task=`df0736e5-f020-4af8-a3ee-9e235128b363`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:06:45,670::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.215459 s, 4.8 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:06:47,690::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:06:47,879::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0884588 s, 11.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:06:48,323::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:06:48,379::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0013893 s, 2.9 MB/s\n'; <rc> = 0
>Thread-16090::DEBUG::2013-03-18 08:06:48,393::task::568::TaskManager.Task::(_updateState) Task=`4d17fc01-bca2-49be-959c-2383c3c1b870`::moving from state init -> state preparing
>Thread-16090::INFO::2013-03-18 08:06:48,394::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-16090::INFO::2013-03-18 08:06:48,394::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0440270900726', 'lastCheck': '3.5', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00904989242554', 'lastCheck': '8.0', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0608651638031', 'lastCheck': '0.0', 'code': 0, 'valid': True}}
>Thread-16090::DEBUG::2013-03-18 08:06:48,395::task::1151::TaskManager.Task::(prepare) Task=`4d17fc01-bca2-49be-959c-2383c3c1b870`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0440270900726', 'lastCheck': '3.5', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00904989242554', 'lastCheck': '8.0', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0608651638031', 'lastCheck': '0.0', 'code': 0, 'valid': True}}
>Thread-16090::DEBUG::2013-03-18 08:06:48,395::task::568::TaskManager.Task::(_updateState) Task=`4d17fc01-bca2-49be-959c-2383c3c1b870`::moving from state preparing -> state finished
>Thread-16090::DEBUG::2013-03-18 08:06:48,395::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16090::DEBUG::2013-03-18 08:06:48,396::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16090::DEBUG::2013-03-18 08:06:48,396::task::957::TaskManager.Task::(_decref) Task=`4d17fc01-bca2-49be-959c-2383c3c1b870`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:06:49,900::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:06:50,055::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0565629 s, 18.1 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:06:52,075::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:06:52,243::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0678798 s, 15.1 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:06:54,265::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:06:54,425::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0586992 s, 17.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:06:54,888::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:06:54,929::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00221664 s, 1.8 MB/s\n'; <rc> = 0
>Thread-16095::DEBUG::2013-03-18 08:06:55,708::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16095::DEBUG::2013-03-18 08:06:55,710::task::568::TaskManager.Task::(_updateState) Task=`f7514215-daa2-48a1-abcc-db334f893ab0`::moving from state init -> state preparing
>Thread-16095::INFO::2013-03-18 08:06:55,711::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16095::INFO::2013-03-18 08:06:55,712::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16095::DEBUG::2013-03-18 08:06:55,712::task::1151::TaskManager.Task::(prepare) Task=`f7514215-daa2-48a1-abcc-db334f893ab0`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16095::DEBUG::2013-03-18 08:06:55,713::task::568::TaskManager.Task::(_updateState) Task=`f7514215-daa2-48a1-abcc-db334f893ab0`::moving from state preparing -> state finished
>Thread-16095::DEBUG::2013-03-18 08:06:55,713::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16095::DEBUG::2013-03-18 08:06:55,713::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16095::DEBUG::2013-03-18 08:06:55,714::task::957::TaskManager.Task::(_decref) Task=`f7514215-daa2-48a1-abcc-db334f893ab0`::ref 0 aborting False
>Thread-16096::DEBUG::2013-03-18 08:06:55,735::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16096::DEBUG::2013-03-18 08:06:55,736::task::568::TaskManager.Task::(_updateState) Task=`a8a3e3d3-f062-44ba-833b-159e553679ac`::moving from state init -> state preparing
>Thread-16096::INFO::2013-03-18 08:06:55,737::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16096::DEBUG::2013-03-18 08:06:55,738::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`8f11df9d-0103-4fec-a645-94c4fd223191`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-16096::DEBUG::2013-03-18 08:06:55,738::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-16096::DEBUG::2013-03-18 08:06:55,739::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-16096::DEBUG::2013-03-18 08:06:55,739::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`8f11df9d-0103-4fec-a645-94c4fd223191`::Granted request
>Thread-16096::DEBUG::2013-03-18 08:06:55,740::task::794::TaskManager.Task::(resourceAcquired) Task=`a8a3e3d3-f062-44ba-833b-159e553679ac`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-16096::DEBUG::2013-03-18 08:06:55,740::task::957::TaskManager.Task::(_decref) Task=`a8a3e3d3-f062-44ba-833b-159e553679ac`::ref 1 aborting False
>Thread-16096::INFO::2013-03-18 08:06:55,743::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16096::DEBUG::2013-03-18 08:06:55,743::task::1151::TaskManager.Task::(prepare) Task=`a8a3e3d3-f062-44ba-833b-159e553679ac`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16096::DEBUG::2013-03-18 08:06:55,743::task::568::TaskManager.Task::(_updateState) Task=`a8a3e3d3-f062-44ba-833b-159e553679ac`::moving from state preparing -> state finished
>Thread-16096::DEBUG::2013-03-18 08:06:55,743::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-16096::DEBUG::2013-03-18 08:06:55,743::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16096::DEBUG::2013-03-18 08:06:55,744::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-16096::DEBUG::2013-03-18 08:06:55,744::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-16096::DEBUG::2013-03-18 08:06:55,744::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-16096::DEBUG::2013-03-18 08:06:55,744::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-16096::DEBUG::2013-03-18 08:06:55,745::task::957::TaskManager.Task::(_decref) Task=`a8a3e3d3-f062-44ba-833b-159e553679ac`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:06:56,445::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:06:56,675::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0792712 s, 12.9 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:06:58,391::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:06:58,439::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00196612 s, 2.1 MB/s\n'; <rc> = 0
>Thread-16098::DEBUG::2013-03-18 08:06:58,668::task::568::TaskManager.Task::(_updateState) Task=`00598883-8d84-4f2c-abc2-257553ec5eb4`::moving from state init -> state preparing
>Thread-16098::INFO::2013-03-18 08:06:58,669::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-16098::INFO::2013-03-18 08:06:58,669::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0437440872192', 'lastCheck': '3.7', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0131521224976', 'lastCheck': '8.2', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0492260456085', 'lastCheck': '0.2', 'code': 0, 'valid': True}}
>Thread-16098::DEBUG::2013-03-18 08:06:58,670::task::1151::TaskManager.Task::(prepare) Task=`00598883-8d84-4f2c-abc2-257553ec5eb4`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0437440872192', 'lastCheck': '3.7', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.0131521224976', 'lastCheck': '8.2', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0492260456085', 'lastCheck': '0.2', 'code': 0, 'valid': True}}
>Thread-16098::DEBUG::2013-03-18 08:06:58,670::task::568::TaskManager.Task::(_updateState) Task=`00598883-8d84-4f2c-abc2-257553ec5eb4`::moving from state preparing -> state finished
>Thread-16098::DEBUG::2013-03-18 08:06:58,670::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16098::DEBUG::2013-03-18 08:06:58,670::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16098::DEBUG::2013-03-18 08:06:58,671::task::957::TaskManager.Task::(_decref) Task=`00598883-8d84-4f2c-abc2-257553ec5eb4`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:06:58,694::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:06:58,880::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.078187 s, 13.1 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:07:00,901::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:07:01,086::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0849375 s, 12.1 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:07:03,106::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:07:03,285::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0761745 s, 13.4 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:07:04,976::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:07:05,019::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.0134319 s, 305 kB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:07:05,307::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:07:05,494::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0888708 s, 11.5 MB/s\n'; <rc> = 0
>Thread-16103::DEBUG::2013-03-18 08:07:05,870::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16103::DEBUG::2013-03-18 08:07:05,872::task::568::TaskManager.Task::(_updateState) Task=`0c6bfe30-747b-4bef-9b4b-a52cb2a6abc5`::moving from state init -> state preparing
>Thread-16103::INFO::2013-03-18 08:07:05,873::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16103::INFO::2013-03-18 08:07:05,874::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16103::DEBUG::2013-03-18 08:07:05,874::task::1151::TaskManager.Task::(prepare) Task=`0c6bfe30-747b-4bef-9b4b-a52cb2a6abc5`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16103::DEBUG::2013-03-18 08:07:05,875::task::568::TaskManager.Task::(_updateState) Task=`0c6bfe30-747b-4bef-9b4b-a52cb2a6abc5`::moving from state preparing -> state finished
>Thread-16103::DEBUG::2013-03-18 08:07:05,875::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16103::DEBUG::2013-03-18 08:07:05,875::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16103::DEBUG::2013-03-18 08:07:05,875::task::957::TaskManager.Task::(_decref) Task=`0c6bfe30-747b-4bef-9b4b-a52cb2a6abc5`::ref 0 aborting False
>Thread-16104::DEBUG::2013-03-18 08:07:05,897::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16104::DEBUG::2013-03-18 08:07:05,898::task::568::TaskManager.Task::(_updateState) Task=`0b3c90fd-f5c8-4ba1-8b60-8caecea9ed3f`::moving from state init -> state preparing
>Thread-16104::INFO::2013-03-18 08:07:05,899::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16104::DEBUG::2013-03-18 08:07:05,900::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`e296ce4d-1f93-4980-ac1d-15584a03738a`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-16104::DEBUG::2013-03-18 08:07:05,901::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-16104::DEBUG::2013-03-18 08:07:05,903::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-16104::DEBUG::2013-03-18 08:07:05,904::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`e296ce4d-1f93-4980-ac1d-15584a03738a`::Granted request
>Thread-16104::DEBUG::2013-03-18 08:07:05,905::task::794::TaskManager.Task::(resourceAcquired) Task=`0b3c90fd-f5c8-4ba1-8b60-8caecea9ed3f`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-16104::DEBUG::2013-03-18 08:07:05,905::task::957::TaskManager.Task::(_decref) Task=`0b3c90fd-f5c8-4ba1-8b60-8caecea9ed3f`::ref 1 aborting False
>Thread-16104::INFO::2013-03-18 08:07:05,909::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16104::DEBUG::2013-03-18 08:07:05,909::task::1151::TaskManager.Task::(prepare) Task=`0b3c90fd-f5c8-4ba1-8b60-8caecea9ed3f`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16104::DEBUG::2013-03-18 08:07:05,910::task::568::TaskManager.Task::(_updateState) Task=`0b3c90fd-f5c8-4ba1-8b60-8caecea9ed3f`::moving from state preparing -> state finished
>Thread-16104::DEBUG::2013-03-18 08:07:05,910::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-16104::DEBUG::2013-03-18 08:07:05,910::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16104::DEBUG::2013-03-18 08:07:05,911::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-16104::DEBUG::2013-03-18 08:07:05,911::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-16104::DEBUG::2013-03-18 08:07:05,912::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-16104::DEBUG::2013-03-18 08:07:05,912::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-16104::DEBUG::2013-03-18 08:07:05,912::task::957::TaskManager.Task::(_decref) Task=`0b3c90fd-f5c8-4ba1-8b60-8caecea9ed3f`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:07:07,514::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:07:07,688::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0751296 s, 13.6 MB/s\n'; <rc> = 0
>Thread-22::DEBUG::2013-03-18 08:07:08,445::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/f33e50cc-eb0a-4720-857d-bc7001a72567/metadata bs=4096 count=1' (cwd None)
>Thread-22::DEBUG::2013-03-18 08:07:08,480::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00159734 s, 2.6 MB/s\n'; <rc> = 0
>Thread-16106::DEBUG::2013-03-18 08:07:08,927::task::568::TaskManager.Task::(_updateState) Task=`c15f8ddd-8edb-470e-8740-5a7179e1f4ea`::moving from state init -> state preparing
>Thread-16106::INFO::2013-03-18 08:07:08,928::logUtils::37::dispatcher::(wrapper) Run and protect: repoStats(options=None)
>Thread-16106::INFO::2013-03-18 08:07:08,928::logUtils::39::dispatcher::(wrapper) Run and protect: repoStats, Return response: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0444581508636', 'lastCheck': '3.9', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00978708267212', 'lastCheck': '8.4', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0367980003357', 'lastCheck': '0.4', 'code': 0, 'valid': True}}
>Thread-16106::DEBUG::2013-03-18 08:07:08,929::task::1151::TaskManager.Task::(prepare) Task=`c15f8ddd-8edb-470e-8740-5a7179e1f4ea`::finished: {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'delay': '0.0444581508636', 'lastCheck': '3.9', 'code': 0, 'valid': True}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'delay': '0.00978708267212', 'lastCheck': '8.4', 'code': 0, 'valid': True}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'delay': '0.0367980003357', 'lastCheck': '0.4', 'code': 0, 'valid': True}}
>Thread-16106::DEBUG::2013-03-18 08:07:08,929::task::568::TaskManager.Task::(_updateState) Task=`c15f8ddd-8edb-470e-8740-5a7179e1f4ea`::moving from state preparing -> state finished
>Thread-16106::DEBUG::2013-03-18 08:07:08,929::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16106::DEBUG::2013-03-18 08:07:08,929::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16106::DEBUG::2013-03-18 08:07:08,930::task::957::TaskManager.Task::(_decref) Task=`c15f8ddd-8edb-470e-8740-5a7179e1f4ea`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:07:09,713::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:07:09,911::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0958387 s, 10.7 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:07:11,931::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:07:12,110::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0783053 s, 13.1 MB/s\n'; <rc> = 0
>Dummy-15786::DEBUG::2013-03-18 08:07:14,130::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)
>Dummy-15786::DEBUG::2013-03-18 08:07:14,338::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0819732 s, 12.5 MB/s\n'; <rc> = 0
>Thread-20::DEBUG::2013-03-18 08:07:15,066::misc::83::Storage.Misc.excCmd::(<lambda>) '/bin/dd iflag=direct if=/dev/4bf968ef-020a-47ee-a1ae-f85eaeb8751a/metadata bs=4096 count=1' (cwd None)
>Thread-20::DEBUG::2013-03-18 08:07:15,095::misc::83::Storage.Misc.excCmd::(<lambda>) SUCCESS: <err> = '1+0 records in\n1+0 records out\n4096 bytes (4.1 kB) copied, 0.00091653 s, 4.5 MB/s\n'; <rc> = 0
>Thread-16111::DEBUG::2013-03-18 08:07:16,028::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16111::DEBUG::2013-03-18 08:07:16,030::task::568::TaskManager.Task::(_updateState) Task=`3c851b1b-8f45-4b9c-aece-baf24ef37973`::moving from state init -> state preparing
>Thread-16111::INFO::2013-03-18 08:07:16,031::logUtils::37::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16111::INFO::2013-03-18 08:07:16,032::logUtils::39::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16111::DEBUG::2013-03-18 08:07:16,033::task::1151::TaskManager.Task::(prepare) Task=`3c851b1b-8f45-4b9c-aece-baf24ef37973`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 1}}
>Thread-16111::DEBUG::2013-03-18 08:07:16,033::task::568::TaskManager.Task::(_updateState) Task=`3c851b1b-8f45-4b9c-aece-baf24ef37973`::moving from state preparing -> state finished
>Thread-16111::DEBUG::2013-03-18 08:07:16,033::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {}
>Thread-16111::DEBUG::2013-03-18 08:07:16,034::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16111::DEBUG::2013-03-18 08:07:16,034::task::957::TaskManager.Task::(_decref) Task=`3c851b1b-8f45-4b9c-aece-baf24ef37973`::ref 0 aborting False
>Thread-16112::DEBUG::2013-03-18 08:07:16,057::BindingXMLRPC::161::vds::(wrapper) [10.34.1.147]
>Thread-16112::DEBUG::2013-03-18 08:07:16,058::task::568::TaskManager.Task::(_updateState) Task=`8f88d825-bab1-4db1-b1fc-a2283e6c56b1`::moving from state init -> state preparing
>Thread-16112::INFO::2013-03-18 08:07:16,058::logUtils::37::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID='a1112cc0-b4af-4396-a3b6-2a3916293a6b', options=None)
>Thread-16112::DEBUG::2013-03-18 08:07:16,060::resourceManager::175::ResourceManager.Request::(__init__) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`6d0d21d7-20d3-4ab5-8f90-864502d210d2`::Request was made in '/usr/share/vdsm/storage/resourceManager.py' line '174' at '__init__'
>Thread-16112::DEBUG::2013-03-18 08:07:16,060::resourceManager::486::ResourceManager::(registerResource) Trying to register resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' for lock type 'shared'
>Thread-16112::DEBUG::2013-03-18 08:07:16,061::resourceManager::528::ResourceManager::(registerResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free. Now locking as 'shared' (1 active user)
>Thread-16112::DEBUG::2013-03-18 08:07:16,061::resourceManager::212::ResourceManager.Request::(grant) ResName=`Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b`ReqID=`6d0d21d7-20d3-4ab5-8f90-864502d210d2`::Granted request
>Thread-16112::DEBUG::2013-03-18 08:07:16,062::task::794::TaskManager.Task::(resourceAcquired) Task=`8f88d825-bab1-4db1-b1fc-a2283e6c56b1`::_resourcesAcquired: Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b (shared)
>Thread-16112::DEBUG::2013-03-18 08:07:16,063::task::957::TaskManager.Task::(_decref) Task=`8f88d825-bab1-4db1-b1fc-a2283e6c56b1`::ref 1 aborting False
>Thread-16112::INFO::2013-03-18 08:07:16,066::logUtils::39::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16112::DEBUG::2013-03-18 08:07:16,066::task::1151::TaskManager.Task::(prepare) Task=`8f88d825-bab1-4db1-b1fc-a2283e6c56b1`::finished: {'info': {'spm_id': 2, 'master_uuid': '4bf968ef-020a-47ee-a1ae-f85eaeb8751a', 'name': 'iscis', 'version': '3', 'domains': u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a:Active,848fa27a-00f3-4bb4-b4da-101cd912b66e:Active,f33e50cc-eb0a-4720-857d-bc7001a72567:Active', 'pool_status': 'connected', 'isoprefix': u'/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/848fa27a-00f3-4bb4-b4da-101cd912b66e/images/11111111-1111-1111-1111-111111111111', 'type': 'ISCSI', 'master_ver': 1, 'lver': 1}, 'dominfo': {u'4bf968ef-020a-47ee-a1ae-f85eaeb8751a': {'status': u'Active', 'diskfree': '6174015488', 'alerts': [], 'disktotal': '10334765056'}, u'848fa27a-00f3-4bb4-b4da-101cd912b66e': {'status': u'Active', 'diskfree': '2337275904', 'alerts': [], 'disktotal': '9314500608'}, u'f33e50cc-eb0a-4720-857d-bc7001a72567': {'status': u'Active', 'diskfree': '44828721152', 'alerts': [], 'disktotal': '53284438016'}}}
>Thread-16112::DEBUG::2013-03-18 08:07:16,066::task::568::TaskManager.Task::(_updateState) Task=`8f88d825-bab1-4db1-b1fc-a2283e6c56b1`::moving from state preparing -> state finished
>Thread-16112::DEBUG::2013-03-18 08:07:16,067::resourceManager::809::ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b': < ResourceRef 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', isValid: 'True' obj: 'None'>}
>Thread-16112::DEBUG::2013-03-18 08:07:16,067::resourceManager::844::ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {}
>Thread-16112::DEBUG::2013-03-18 08:07:16,067::resourceManager::538::ResourceManager::(releaseResource) Trying to release resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b'
>Thread-16112::DEBUG::2013-03-18 08:07:16,068::resourceManager::553::ResourceManager::(releaseResource) Released resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' (0 active users)
>Thread-16112::DEBUG::2013-03-18 08:07:16,068::resourceManager::558::ResourceManager::(releaseResource) Resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b' is free, finding out if anyone is waiting for it.
>Thread-16112::DEBUG::2013-03-18 08:07:16,069::resourceManager::565::ResourceManager::(releaseResource) No one is waiting for resource 'Storage.a1112cc0-b4af-4396-a3b6-2a3916293a6b', Clearing records.
>Thread-16112::DEBUG::2013-03-18 08:07:16,069::task::957::TaskManager.Task::(_decref) Task=`8f88d825-bab1-4db1-b1fc-a2283e6c56b1`::ref 0 aborting False
>Dummy-15786::DEBUG::2013-03-18 08:07:16,369::misc::83::Storage.Misc.excCmd::(<lambda>) 'dd if=/rhev/data-center/a1112cc0-b4af-4396-a3b6-2a3916293a6b/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000' (cwd None)

Actions: View

Attachments on bug 921643: 711787 | 711788